Trouble Accessing Document ID in WeaviateHybridSearchRetriever Results
Labels: name: area: vector store, name: auto:bug

### System Info

langchain: 0.0.334
python: 3.11.6
weaviate-client: 3.25.3

### Who can help?

_No response_

### Information

- [X] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

I am trying to implement the WeaviateHybridSearchRetriever to retrieve documents from Weaviate. My schema indicates the document ID is stored in the _id field based on the shardingConfig.

When setting up the retriever, I included _id in the attributes list:
````
hybrid_retriever = WeaviateHybridSearchRetriever(
  attributes=["_id", "aliases", "categoryid", "name", "page_content", "ticker"]
)
````
However, when I try to access _id on the returned Document objects, I get an error that _id is not found.

For example:

````
results = hybrid_retriever.get_relevant_documents(query="some query")
print(results[0]._id) # Error!_id not found
````

I have tried variations like id, document_id instead of _id but still cannot seem to access the document ID field.


Any suggestions on what I am missing or doing wrong when trying to retrieve the document ID from the Weaviate results using the _id field specified in the schema?

Let me know if any other details would be helpful in troubleshooting this issue!

**Schema Details**
````
{
   "classes":[
      {
         "class":"Category_taxonomy",
         "invertedIndexConfig":{
            "bm25":{
               "b":0.75,
               "k1":1.2
            },
            "cleanupIntervalSeconds":60,
            "stopwords":{
               "additions":"None",
               "preset":"en",
               "removals":"None"
            }
         },
         "moduleConfig":{
            "text2vec-openai":{
               "baseURL":"https://api.openai.com",
               "model":"ada",
               "modelVersion":"002",
               "type":"text",
               "vectorizeClassName":true
            }
         },
         "multiTenancyConfig":{
            "enabled":false
         },
         "properties":[
            {
               "dataType":[
                  "text"
               ],
               "description":"Content of the page",
               "indexFilterable":true,
               "indexSearchable":true,
               "moduleConfig":{
                  "text2vec-openai":{
                     "skip":false,
                     "vectorizePropertyName":false
                  }
               },
               "name":"page_content",
               "tokenization":"word"
            },
            {
               "dataType":[
                  "number"
               ],
               "description":"Identifier for the category",
               "indexFilterable":true,
               "indexSearchable":false,
               "moduleConfig":{
                  "text2vec-openai":{
                     "skip":false,
                     "vectorizePropertyName":false
                  }
               },
               "name":"categoryid"
            },
            {
               "dataType":[
                  "text"
               ],
               "description":"Ticker symbol",
               "indexFilterable":true,
               "indexSearchable":true,
               "moduleConfig":{
                  "text2vec-openai":{
                     "skip":false,
                     "vectorizePropertyName":false
                  }
               },
               "name":"ticker",
               "tokenization":"word"
            },
            {
               "dataType":[
                  "text"
               ],
               "description":"Name of the entity",
               "indexFilterable":true,
               "indexSearchable":true,
               "moduleConfig":{
                  "text2vec-openai":{
                     "skip":false,
                     "vectorizePropertyName":false
                  }
               },
               "name":"name",
               "tokenization":"word"
            },
            {
               "dataType":[
                  "text"
               ],
               "description":"Aliases for the entity",
               "indexFilterable":true,
               "indexSearchable":true,
               "moduleConfig":{
                  "text2vec-openai":{
                     "skip":false,
                     "vectorizePropertyName":false
                  }
               },
               "name":"aliases",
               "tokenization":"word"
            }
         ],
         "replicationConfig":{
            "factor":1
         },
         "shardingConfig":{
            "virtualPerPhysical":128,
            "desiredCount":1,
            "actualCount":1,
            "desiredVirtualCount":128,
            "actualVirtualCount":128,
            "key":"_id",
            "strategy":"hash",
            "function":"murmur3"
         },
         "vectorIndexConfig":{
            "skip":false,
            "cleanupIntervalSeconds":300,
            "maxConnections":64,
            "efConstruction":128,
            "ef":-1,
            "dynamicEfMin":100,
            "dynamicEfMax":500,
            "dynamicEfFactor":8,
            "vectorCacheMaxObjects":1000000000000,
            "flatSearchCutoff":40000,
            "distance":"cosine",
            "pq":{
               "enabled":false,
               "bitCompression":false,
               "segments":0,
               "centroids":256,
               "trainingLimit":100000,
               "encoder":{
                  "type":"kmeans",
                  "distribution":"log-normal"
               }
            }
         },
         "vectorIndexType":"hnsw",
         "vectorizer":"text2vec-openai"
      }
   ]
}
````

**Example Document**
````
{
    "class": "Category_taxonomy",
    "creationTimeUnix": 1699553747601,
    "id": "ad092eb1-e4a6-4d93-a7d2-c507c33c3837",
    "lastUpdateTimeUnix": 1699553747601,
    "properties": {
        "aliases": "Binance Coin, Binance Smart Chain",
        "categoryid": 569,
        "name": "BNB",
        "page_content": "ticker: bnb\nname: BNB\naliases: Binance Coin, Binance Smart Chain",
        "ticker": "bnb"
    },
    "vectorWeights": null
}
````

Example Search Result
````
{
   "status":"success",
   "results":[
      {
         "page_content":"ticker: bnb\nname: BNB\naliases: Binance Coin, Binance Smart Chain",
         "metadata":{
            "_additional":{
               "explainScore":"(vector) [-0.0067740963 -0.03091735 0.00511335 0.0016186031 -0.016120477 0.017543973 -0.0072548385 -0.023063144 0.015246399 -0.0020884196]...  \n(hybrid) Document ad092eb1-e4a6-4d93-a7d2-c507c33c3837 contributed 0.00819672131147541 to the score",
               "score":"0.008196721"
            },
            "aliases":"Binance Coin, Binance Smart Chain",
            "categoryid":569,
            "name":"BNB",
            "ticker":"bnb"
         },
         "type":"Document"
      }
   ]
}
````

App Code

````
# Prepare global variables
WEAVIATE_URL = os.getenv('WEAVIATE_URL')
WEAVIATE_API_KEY = os.getenv('WEAVIATE_API_KEY')
OPENAI_API_KEY = os.getenv('OPENAI_API_KEY')
INDEX_NAME = "Category_taxonomy"
TEXT_KEY = "page_content"

# Dependency provider function for Weaviate client
def get_weaviate_vectorstore():
    # Initialize the Weaviate client with API key authentication
    client = weaviate.Client(
        url=WEAVIATE_URL, 
        auth_client_secret=weaviate.AuthApiKey(WEAVIATE_API_KEY),
        additional_headers={
            "X-Openai-Api-Key": OPENAI_API_KEY,
        }
    )

    # Initialize embeddings with a specified model
    embeddings = OpenAIEmbeddings(openai_api_key=OPENAI_API_KEY, model='text-embedding-ada-002')

    # Initialize vector store with attributes and schema
    vectorstore = Weaviate(
        client=client,
        index_name=INDEX_NAME,
        text_key=TEXT_KEY,
        embedding=embeddings,
        attributes=["aliases", "categoryid", "name", "page_content", "ticker"],
        by_text=False
    )
    return client, vectorstore

def get_weaviate_hybrid_retriever(k: int = 5):
    # Directly call the function to get the client and vectorstore
    client, vectorstore = get_weaviate_vectorstore()
    
    # Instantiate the retriever with the settings from the vectorstore
    hybrid_retriever = WeaviateHybridSearchRetriever(
        client=client,
        index_name=INDEX_NAME,
        text_key=TEXT_KEY,
        attributes=["aliases", "categoryid", "name", "page_content", "ticker"],
        k=k,
        create_schema_if_missing=True
    )
    return hybrid_retriever

async def parse_query_params(request: Request) -> Dict[str, List[Any]]:
    parsed_values = defaultdict(list)

    for key, value in request.query_params.multi_items():
        # Append the value for any key directly
        parsed_values[key].append(value)

    return parsed_values

@router.get("/hybrid_search_category_taxonomy/")
async def hybrid_search_category_taxonomy(parsed_values: Dict[str, List[Any]] = Depends(parse_query_params), query: Optional[str] = None, k: int = 5):
    categoryids = parsed_values.get('categoryid', [])
    tickers = parsed_values.get('ticker', [])
    names = parsed_values.get('name', [])
    aliasess = parsed_values.get('aliases', [])

    # Use a partial function to pass 'k' to 'get_weaviate_hybrid_retriever'
    retriever = get_weaviate_hybrid_retriever(k=k)
    
    # Initialize the where_filter with an 'And' operator if there are any filters provided
    logging.info(
        f"query: {query}, "
        f"categoryID: {categoryids}, "
        f"ticker: {tickers}, "
        f"name: {names}, "
        f"aliases: {aliasess}, "
        f"k: {k}"
    )

    # Adjustments to reference parameters from 'parse_query_params'
    where_filter = {"operator": "And", "operands": []} if any([categoryids, tickers, names, aliasess]) else None
    
    # Add filters for categoryid and ticker with the 'Equal' operator
    if categoryids:
        category_operands = [{"path": ["categoryid"], "operator": "Equal", "valueNumber": cid} for cid in categoryids]
        if category_operands:
            where_filter["operands"].append({"operator": "Or", "operands": category_operands})
        
    if tickers:
        ticker_operands = [{"path": ["ticker"], "operator": "Equal", "valueText": ticker} for ticker in tickers]
        if ticker_operands:
            where_filter["operands"].append({"operator": "Or", "operands": ticker_operands})
    
    if names:
        name_operands = [{"path": ["name"], "operator": "Equal", "valueText": name} for name in names]
        if name_operands:
            where_filter["operands"].append({"operator": "Or", "operands": name_operands})
    
    if aliasess:
        aliases_operands = [{"path": ["aliases"], "operator": "Equal", "valueText": aliases} for aliases in aliasess]
        if aliases_operands:
            where_filter["operands"].append({"operator": "Or", "operands": aliases_operands})
    
    try:
        # Format the results for the response
        effective_query = " " if not query or not query.strip() else query

        # Log the where_filter before fetching documents
        logging.info(f"where_filter being used: {where_filter}")
        
        # Fetch the relevant documents using the hybrid retriever instance
        results = retriever.get_relevant_documents(effective_query, where_filter=where_filter, score=True)

        # Format the results for the response
        response_data = [vars(doc) for doc in results]
        
        return {"status": "success", "results": response_data}
    except Exception as e:
        logger.error(f"Error while processing request: {str(e)}", exc_info=True)
        raise HTTPException(detail=str(e), status_code=500)
````

### Expected behavior

**Expected Behavior**

When using the WeaviateHybridSearchRetriever for document retrieval, I expect that including the _id attribute in the attributes list will allow me to access the document ID of each retrieved document without any issues. Specifically, after setting up the WeaviateHybridSearchRetriever like so:

````
hybrid_retriever = WeaviateHybridSearchRetriever(
    attributes=["_id", "aliases", "categoryid", "name", "page_content", "ticker"]
)
````

I anticipate that executing a query and attempting to print the _id of the first result should successfully return the unique identifier of the document, as per the below code snippet:

````
results = hybrid_retriever.get_relevant_documents(query="some query")
print(results[0]._id)  # Expecting this to print the _id of the first result
````

In this scenario, my expectation is that the _id field, being specified in the attributes parameter, should be readily accessible in each Document object returned by the get_relevant_documents method. This behavior is crucial for my application as it relies on the unique document IDs for further processing and analysis of the retrieved data.