Add Language Agnostic Bert Sentence Embedding
Author: alexcleu
Pull request: no
Labels: name: wontfix

name: New model

# 🌟 New model addition

## Model description
Google released a new model LaBSe, which is a Language agnostic Bert sentence embedding with 109 language support. 

https://ai.googleblog.com/2020/08/language-agnostic-bert-sentence.html?utm_source=Deep+Learning+Weekly&utm_campaign=cdb8afedb7-EMAIL_CAMPAIGN_2019_04_24_03_18_COPY_01&utm_medium=email&utm_term=0_384567b42d-cdb8afedb7-72940109

The model is also hosted on tfhub as well:https://tfhub.dev/google/LaBSE/1

<!-- Important information -->

## Open source status

* [ ] the model implementation is available: (https://arxiv.org/abs/2007.01852)
* [ ] the model weights are available: https://tfhub.dev/google/LaBSE/1
* [ ] who are the authors: (mention them, if possible by @gh-username)


Comments:
Any update on this if this will be taken up by the team?

I think this is already done right? seen that we have this https://github.com/huggingface/transformers/tree/master/model_cards/pvl/labse_bert

The model uploaded by @pvl mentioned by @aalloul  performs the wrong pooling, i.e., embeddings produced by that model are NOT the same as the embeddings from the TFHub version.

I uploaded the model with the right pooling here:
https://huggingface.co/sentence-transformers/LaBSE

It tested it against the TF Hub version and it produces similar embeddings (small epsilon difference due to different padding variations). On down stream tasks, the TFHub and the HF Pytorch version achieve the same performance.

For LABSE model can we use FAISS to build parallel corpus. If not what is the best best algo to that. LABSE paper was suggesting ANN. Where can I find its implementation for building parallel corpus.

Have a look here for some examples with ANN, including FAISS
https://github.com/UKPLab/sentence-transformers/tree/master/examples/applications

Personally I prefer hnswlib over Faiss:
https://github.com/UKPLab/sentence-transformers/blob/master/examples/applications/semantic_search_quora_hnswlib.py


It is nicer and easier to use, better documented and offer certain features that are missing in Faiss. FAISS added later hnswlib as index structure, as it is also faster than the other index types FAISS were offering.

Yes, you can use LaBSE with ANN. 

@nreimers Which ANN LaBSE paper was recommending?

@aj7tesh I think they are not mentioning which ANN is used.

As it is a google paper, I could imagine they use SCANN:
https://github.com/google-research/google-research/tree/master/scann

A good source for comparison is:
http://ann-benchmarks.com/index.html

Here, HNSWLib performs really well:
https://github.com/nmslib/hnswlib

@nreimers thank you. Surely I will explore above comparisons. Will see which one is helping me more to generate parallel corpus.

32GB should be more than fine. Issues can be:
- too long sequences. Try to use max_length in the tokenizer
- Too large batch sizes. Try to use a smaller batch size.


basically
# in my case i have lets say more than 2k sentences in array
# its passing the encoded_input step, however its going OOM in model_output. Usually on my machine its works fine for upto 10k sentences when using LASER, however for LABSE its failing after 150 only
encoded_input = tokenizer(sentences, padding=True, truncation=True, max_length=64, return_tensors='pt')

with torch.no_grad():
    model_output = model(**encoded_input, return_dict=True)

@nreimers 
its working fine when I used below method

from sentence_transformers import SentenceTransformer

not exaclty sure the reason(hope model weights are similar to tf model)

I have a questions, does these Labse and Laser kind of multilingual model works on language which is not related to major languages on which these models are trained? I believe for zero shot learning the language should have some similarity to other major languages. 

sentence transformers performs batching of your data. If you pass 10k sentences, it splits it into batches of e.g. 32 and encodes them. So that you don't run out of memory.

Embeddings are nearly identical to those of Tensorflow. I tested both models on Tatoeba test set on 100 languages, and Pytorch and TF Version perform equally.


Depends on the language. If the language is really similar to another language that was used for training, then it works. If it uses a different vocab or is really different, then it doesn't work.

> Have a look here for some examples with ANN, including FAISS
> https://github.com/UKPLab/sentence-transformers/tree/master/examples/applications
> 
> Personally I prefer hnswlib over Faiss:
> https://github.com/UKPLab/sentence-transformers/blob/master/examples/applications/semantic_search_quora_hnswlib.py
> 
> It is nicer and easier to use, better documented and offer certain features that are missing in Faiss. FAISS added later hnswlib as index structure, as it is also faster than the other index types FAISS were offering.
> 
> Yes, you can use LaBSE with ANN.

I was going through this HNSW implementation. At some places it was written that ANN is not perfect and then HNSWs results were compared against util.semantic search, which again was executing cosine similarity for bitext mining. What is the reason of performing this step.
I understand this thread is not the right place to ask this question, kindly suggest some other thread or place for such queries

Approximate Nearest Neighbor only returns approximately the 10 nearest neighbor. It can be that it misses points that are closer. This is expressed as recall. 

In the examples, it compares ANN against an exact nearest neighbor to see if there might be an issue with the index construction.

For large datasets, exact search is too slow, so you have to live with it that ANN does not find perfectly all nearest neighbors.

so for corpus building where I expect the sentence with highest cosine similarity be the translation pair for corresponding source sentence. I will have to go for exact matches using something like util.semantic_search or scipy spatial distance

If the corpora are not too large, yes, you can use exact matches. But these methods have quadratic runtime, i.e., when you have 10 Millions of sentences, searching will take a long time.

If your corpus is smaller, you can use exact search.

Got it @nreimers  thanks

> The model uploaded by @pvl mentioned by @aalloul performs the wrong pooling, i.e., embeddings produced by that model are NOT the same as the embeddings from the TFHub version.
> 
> I uploaded the model with the right pooling here:
> https://huggingface.co/sentence-transformers/LaBSE
> 
> It tested it against the TF Hub version and it produces similar embeddings (small epsilon difference due to different padding variations). On down stream tasks, the TFHub and the HF Pytorch version achieve the same performance.

@nreimers I'm guessing there should be only 1 implementation of LaBSE and people might get confused with which one to use. How should we go about this?

Coming late to this thread, we also uploaded a pytorch and TF compatible versions of the LaBSE model here - https://huggingface.co/rasa/LaBSE . This will also be available inside Rasa Open Source very soon.
I do agree with @aalloul about the confusion this can create. Looking for thoughts from folks on this.

@dakshvar22 did you run any comparisons with the official model?

> I'm guessing there should be only 1 implementation of LaBSE and people might get confused with which one to use. How should we go about this?

We could imagine building a curation system built on top of (e.g.) a combination of downloads and an explicit marker like a "Star" button, but I don't want to overfit too much to the first few examples – given that this use case is still not super frequent.

Happy to hear anyone's thoughts on this

@aalloul I cross-checked the embeddings from the TFhub version and the transformers compatible versions we uploaded and they are almost identical. This was on a corpus of around 50k sentences across 5 different languages. Please feel free to test them out on the Tatoeba dataset on all 100 languages. I might not be able to do that myself right now.

@aalloul @dakshvar22 
I tested https://huggingface.co/sentence-transformers/LaBSE on the Tatoeba dataset on all 100+ languages and the performances were comparable to the TFHub model (+/- 0.1 accuracy for some languages due to different padding and numerical stability in pytorch vs. tensorflow)

ah nice, thanks @nreimers for letting us know! I'll have a look at it.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Late to this thread (and noticed it existed after publishing the model), but I ported the model from TF Hub and uploaded it here: https://huggingface.co/setu4993/LaBSE

Additionally, my code to port it, alongside tests that verify the embeddings generated by the source TF Hub model and the ported PyTorch model (uploaded above) are in my repo: https://github.com/setu4993/convert-labse-tf-pt

Should be easy to extend it / add other tests and verify the embeddings match, if someone is interested. I haven't run tests on downstream performance, though.