Repository: dmlc/gluon-nlp

Issue:
Add pre-trained source cc.en.300 for token embedding fasttext
FaceBook Research recently released the English source files.
https://github.com/facebookresearch/fastText/commit/5bf8b4c615b6308d76ad39a5a50fa6c4174113ea
Could those be added to Gluon-NLP as well?



Discussion:
Yes, I can add the English vectors and crawl-300d-2M-subword.zip
Thank you very much! That will be greatly appreciated.
Also, at the moment it's not possible to use _wiki-news-300d-1M-subword.vec.zip_ as source with _load_ngrams=True_
Do I understand it correctly that this should be possible since it has been trained with subword information?
on english, i have observed wiki.en.vec to be much superior in performance over the common crawl embeddings, for other languages that arent data rich, the common crawl ones are doing good.

https://arxiv.org/pdf/1802.06893.pdf

Inline with what the common-crawl embeddings paper suggests
@KurtJanssensCRW unfortunately no. While those vectors were supposedly trained with subword information, the subword part of the embeddings was not published.

If you find the subword part (included in the `.bin` file), we'd be happy to add it. 
@szha could you copy the following files to the GluonNLP bucket

https://lllausen-data.s3.amazonaws.com/gluonnlp/crawl-300d-2M-subword-e6b07293f.bin
https://lllausen-data.s3.amazonaws.com/gluonnlp/crawl-300d-2M-subword-927782c8e.npz

Thanks!

Resolving patch:
diff --git a/gluonnlp/_constants.py b/gluonnlp/_constants.py
--- a/gluonnlp/_constants.py
+++ b/gluonnlp/_constants.py
@@ -73,6 +73,8 @@
 
 FAST_TEXT_NPZ_SHA1 = {
     'crawl-300d-2M': ('crawl-300d-2M.npz', '9dd611a1fe280c63050cd546d3595400fc0eede4'),
+    'crawl-300d-2M-subword': ('crawl-300d-2M-subword-927782c8e.npz',
+                              '927782c8ec8c2c1deb6a8a550217478e183ca25a'),
     'wiki.aa': ('wiki.aa.npz', '48f163b80eb37f1806142169d3d4c05cf75b7339'),
     'wiki.ab': ('wiki.ab.npz', '860ceff119dd27e5b701b605879037c1310cbc3e'),
     'wiki.ace': ('wiki.ace.npz', '62938287464040491719f56a6f521f8f808beee8'),
@@ -561,6 +563,8 @@
 
 
 FAST_TEXT_BIN_SHA1 = {
+    'crawl-300d-2M-subword': ('crawl-300d-2M-subword-e6b07293f.bin',
+                              'e6b07293f7b0095e3c72c2a12bc09464b69444b0'),
     'cc.af.300': ('cc.af.300-33115ff8.bin', '33115ff8e4c8f439757c819399177f1f58f07f12'),
     'cc.als.300': ('cc.als.300-d6579933.bin', 'd65799331a03895d68a3fbe7611b181d7e7cc916'),
     'cc.am.300': ('cc.am.300-999b3e95.bin', '999b3e95a2c490d7fcab2a6e08074746303d3c17'),


Test patch:
diff --git a/tests/unittest/test_vocab_embed.py b/tests/unittest/test_vocab_embed.py
--- a/tests/unittest/test_vocab_embed.py
+++ b/tests/unittest/test_vocab_embed.py
@@ -462,14 +462,14 @@ def test_token_embedding_from_file(tmpdir, allow_extend):
 
 
 def test_embedding_get_and_pretrain_file_names():
-    assert len(nlp.embedding.list_sources(embedding_name='fasttext')) == 484
+    assert len(nlp.embedding.list_sources(embedding_name='fasttext')) == 485
     assert len(nlp.embedding.list_sources(embedding_name='glove')) == 10
     assert len(nlp.embedding.list_sources(embedding_name='word2vec')) == 3
 
     reg = nlp.embedding.list_sources(embedding_name=None)
 
     assert len(reg['glove']) == 10
-    assert len(reg['fasttext']) == 484
+    assert len(reg['fasttext']) == 485
     assert len(reg['word2vec']) == 3
 
     with pytest.raises(KeyError):