Enhance retrieval callback hooks to include information on cosine similarity scores or other retrieval metrics
Labels: name: area: vector store, name: auto:enhancement

### Feature request

Make cosine similarity (or other retrieval metric, e.g., Euclidean distance) between query and retrieved documents available at the `retriever` callback hook.

Currently, these scores are computed during retrieval and are discarded before they become available to the retrieval callback hook:

https://github.com/langchain-ai/langchain/blob/125ae6d9dec440e04a60f63f0f4fc8411f482df8/libs/langchain/langchain/vectorstores/base.py#L500

https://github.com/langchain-ai/langchain/blob/125ae6d9dec440e04a60f63f0f4fc8411f482df8/libs/langchain/langchain/schema/retriever.py#L174

### Motivation

I want to build a callback handler that enables LangChain users to visualize their data in [Phoenix](https://github.com/Arize-ai/phoenix), an open-source tool that provides debugging workflows for retrieval-augmented generation. At the moment, it is not possible to get the similarity scores between queries and retrieved documents out of LangChain's callback system, for example, when using the `RetrievalQA` chain. Here is an [example notebook](https://github.com/Arize-ai/phoenix/blob/main/tutorials/langchain_pinecone_search_and_retrieval_tutorial.ipynb) where I sub-class `Pinecone` to get out the similarity scores:

```
class PineconeWrapper(Pinecone):
    query_text_to_document_score_tuples: Dict[str, List[Tuple[Document, float]]] = {}

    def similarity_search_with_score(
        self,
        query: str,
        k: int = 4,
        filter: Optional[dict] = None,
        namespace: Optional[str] = None,
    ) -> List[Tuple[Document, float]]:
        document_score_tuples = super().similarity_search_with_score(
            query=query,
            k=k,
            filter=filter,
            namespace=namespace,
        )
        self.query_text_to_document_score_tuples[query] = document_score_tuples
        return document_score_tuples

    @property
    def retrieval_dataframe(self) -> pd.DataFrame:
        query_texts = []
        document_texts = []
        retrieval_ranks = []
        scores = []
        for query_text, document_score_tuples in self.query_text_to_document_score_tuples.items():
            for retrieval_rank, (document, score) in enumerate(document_score_tuples):
                query_texts.append(query_text)
                document_texts.append(document.page_content)
                retrieval_ranks.append(retrieval_rank)
                scores.append(score)
        return pd.DataFrame.from_dict(
            {
                "query_text": query_texts,
                "document_text": document_texts,
                "retrieval_rank": retrieval_ranks,
                "score": scores,
            }
        )
```

I would like the LangChain callback system to support this use-case.

### Your contribution

I am willing to implement, test, and document this feature with guidance from the LangChain team. I am also happy to provide feedback on an implementation by the LangChain team by building an example callback handler using the enhancement retrieval hook functionality.