Model Card for EQuIP-Queries/EQuIP_3B

An AI model that understands natural language and translates it into accurate Elasticsearch queries. This model is based on the Qwen2.5 3B architecture, a compact yet powerful language model known for its efficiency. We fine-tuned this model with 10,000 Elasticsearch query data points to specialize its ability to generate accurate and relevant queries.

Model Details

Model Description

Our Solution: An AI-Powered Query Generator Our team has developed a solution to this challenge: an AI model that understands natural language and translates it into accurate Elasticsearch queries. This model is based on the Qwen2.5 3B architecture, a compact yet powerful language model known for its efficiency. We fine-tuned this model with 10,000 Elasticsearch query data points to specialize its ability to generate accurate and relevant queries. We've employed advanced techniques, including LoRA (Low-Rank Adaptation) to optimize the model for performance and efficiency. Specifically, LoRA reduces the number of trainable parameters by introducing low-rank matrices. This combination allows us to achieve high accuracy while minimizing computational resource requirements.

Key Features and Benefits

Natural Language Interface: Users can simply describe the data they're looking for in plain English, and the model will generate the corresponding Elasticsearch query. Increased Efficiency: Reduces the time and effort required to write complex queries, allowing users to focus on analyzing their data. Improved Accessibility: Makes Elasticsearch more accessible to a wider audience, including those who are not experts in its query language. Open Source: We are committed to open source and believe in the power of community-driven innovation. By making our model open source, we aim to contribute to the advancement of AI and empower others to build upon our work. We recognize the lack of readily available solutions in this specific area, and we're excited to fill that gap. Future Developments: This is just the beginning. Our team is dedicated to pushing the boundaries of what's possible with AI, and we have plans to release further updates and enhancements to this model in the future. We are committed to continuous improvement and innovation in the field of AI-powered search. This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.

Developed by: EQuIP
Funded by : EQuIP
Model type: Causal Language Model
Language(s) (NLP): English (en)
License: MIT License
Finetuned from model : Qwen2.5-3B-Instruct

Model Sources [optional]

Repository: https://huggingface.co/EQuIP-Queries/EQuIP_3B

Uses

Direct Use

This model is intended to be directly used to translate natural language prompts into Elasticsearch queries without additional fine-tuning.

Example use cases include:

Generating Elasticsearch queries from plain English prompts. Simplifying query generation for analysts, developers, or data scientists unfamiliar with Elasticsearch syntax. Automating query creation as part of search, analytics, or data exploration tools.

Intended users: Developers integrating natural language querying capabilities into Elasticsearch-based applications. Analysts and data scientists who frequently interact with Elasticsearch data.

Out-of-Scope Use

The model is not intended for use cases such as:

Generating queries for databases or search engines other than Elasticsearch. Handling languages other than English. Providing factual answers or general conversational interactions. Tasks involving sensitive decision-making, such as medical, legal, or financial advice, where inaccurate queries may lead to significant consequences.

Bias, Risks, and Limitations

Bias Awareness:

The model may inherit biases present in the training data. Users should assess generated outputs for unintended biases or patterns, particularly in sensitive contexts.

Misuse and Malicious Use:

Users must avoid using the model to intentionally produce harmful or misleading search queries or manipulate search results negatively.

Limitations:

Performance may degrade significantly if input prompts differ substantially from the fine-tuning data domain.
The model does not validate query accuracy or safety and should be reviewed before execution, especially in production environments.

Recommendations

Query Validation:

Always validate and test generated Elasticsearch queries before deploying in production or using on sensitive data. Automatic generation may occasionally result in syntactic or semantic inaccuracies. Bias Awareness:
The model may inherit biases present in the training data. Users should assess generated outputs for unintended biases or patterns, particularly in sensitive contexts. Use in Sensitive Contexts:
Avoid using this model for critical or high-stakes decision-making tasks without additional human oversight and validation. Continuous Monitoring:
Monitor the outputs regularly to identify and correct issues promptly, ensuring long-term reliability. Transparency:
Clearly communicate the AI-driven nature of generated Elasticsearch queries to end-users to manage expectations and encourage verification.

How to Get Started with the Model

Install the required dependencies:

[pip install transformers torch]

Here's how you can quickly start generating Elasticsearch queries from natural language prompts using this model:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "EQuIP-Queries/EQuIP_3B"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

mapping = "[Your Elasticsearch mappings]"
user_request = "Find me products which are less than $50"

prompt = f"Given this mapping: {mapping}\nGenerate an Elasticsearch query for: {user_request}"

inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(
    inputs["input_ids"],
    max_length=512,
    do_sample=True,
    temperature=0.2,
    top_p=0.9,
    pad_token_id=tokenizer.pad_token_id
)

generated_query = tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print("Generated Elasticsearch query:")
print(generated_query)

Training Details

Training Data

The model was fine-tuned on a custom dataset consisting of 10,000 pairs of natural language prompts and corresponding Elasticsearch queries. Each prompt describes the desired Elasticsearch query in plain English, paired with a manually crafted accurate Elasticsearch query.

The dataset covers various query types and common Elasticsearch query patterns, including filters, range queries, aggregations, boolean conditions, and text search scenarios.

Currently, the dataset is not publicly available. If made available in the future, a Dataset Card link will be provided here.

Preprocessing:

Prompts and queries were cleaned to ensure consistent formatting.
Special tokens and unnecessary whitespace were removed to ensure high-quality training data.

Training Procedure

The model was fine-tuned using Low-Rank Adaptation (LoRA) on top of the pre-trained Qwen2.5-3B-Instruct model. LoRA significantly reduced computational requirements by training only low-rank matrices within the Transformer layers.

Training Hyperparameters

Training regime: bf16 non-mixed precision

Evaluation

The model was evaluated using a held-out test set comprising 1,000 prompt-query pairs not included in the training dataset. The primary goal of the evaluation was to measure the accuracy and relevance of generated Elasticsearch queries.

Testing Data, Factors & Metrics

Testing Data

Size: 1,000 prompt-query pairs (held-out from training).
Composition: Representative of diverse Elasticsearch query types, including boolean conditions, aggregations, text search, and date-based queries.

Factors

The evaluation considered:

Complexity of the Elasticsearch query.
Accuracy in interpreting the intent of natural language prompts.
Syntactic correctness and relevance of generated queries.

Metrics

Exact Match: Measures the percentage of queries matching exactly with ground truth queries. Semantic Similarity: Assessed using embedding-based similarity scores (e.g., cosine similarity). Token-level F1: Evaluates precision and recall at the token-level, measuring partial correctness in generated queries.

Results

Model	Parameters	Generation Time (sec)	Token Precision	Token Recall	Token F1	Validity Rate	Field Similarity
EQuIP	3B	0.7969	0.8738	0.9737	0.9808	0.97	0.9916
LLaMA 3.1	8B	13.4822	0.3979	0.6	0.5693	0.5723	0.4622
Qwen 2.5	7B	1.4233	0.6667	0.7	0.7743	0.82	0.6479
Deepseek Distill	8B	9.2516	0.5846	0.65	0.6979	0.7496	0.8908
Gemma 2	9B	3.0801	0.6786	0.82	0.7309	0.8	0.8151
Mistral	7B	2.1068	0.6786	0.79	0.7551	0.8	0.7437

Summary

The evaluation demonstrates that the model achieves strong performance in accurately translating natural language prompts into valid Elasticsearch queries. It shows particularly high effectiveness in terms of token precision, recall, and overall semantic similarity, highlighting its ability to generate accurate, relevant, and syntactically correct queries efficiently. Compared to several other widely-used models, it offers an excellent balance of accuracy, speed, and computational efficiency, making it highly suitable for production use in Elasticsearch query generation tasks. However, it's recommended that users continue to verify query outputs, especially for critical or sensitive applications.

Environmental Impact

Carbon emissions for the training and fine-tuning of this model can be estimated using the Machine Learning Impact calculator introduced by Lacoste et al. (2019).

Hardware Type: NVIDIA A100 GPU
Hours used: 11 hours
Cloud Provider: Vast.ai

Model Architecture and Objective

This model is based on the Qwen2.5-3B-Instruct architecture, which is a decoder-only, transformer-based causal language model. It consists of approximately 3 billion parameters designed for efficient and high-quality natural language understanding and generation.

The primary objective of this fine-tuned model is to accurately convert natural language prompts into syntactically correct and semantically relevant Elasticsearch queries. To achieve this, the model was fine-tuned on domain-specific data, incorporating Low-Rank Adaptation (LoRA) to optimize performance and resource efficiency.

Model Card Contact

Contact: EQuIP Email: [info@equipqueries.com]

EQuIP-Queries
/

EQuIP_3B