Package: text-sim
Text-SimilarityOverviewDataset: 中文/English 语料, ☞点这里Paper: 相关论文详解, ☞点这里The implemented method is as follows:：TF-IDFBM25LSHSIF/uSIFFastTextRNN Base (Siamese RNN, Stack RNN)CNN Base (Fast Text, Text CNN, Char CNN, VDCNN)Bert BaseAlbertNEZHARoBERTaSimCSEPoly-EncoderColBERTRE2（Simple-Effective-Text-Matching）Usages1：examples目录下有不同模型对应的 preprocess/train/evalute代码，可自行修改
2：如下示例从examples中引入actuator方法，准备好对应的模型配置文件即可执行
3：examples目录下的inference.py为训练好的模型推理代码TF-IDF# Example# Sklearn versionfromexamples.run_tfidf_sklearnimportactuatoractuator("./corpus/chinese/breeno/train.tsv",query1="12 23 4160 276",query2="29 23 169 1495")# Custom versionfromexamples.run_tfidfimportactuatoractuator("./corpus/chinese/breeno/train.tsv",query1="12 23 4160 276",query2="29 23 169 1495")# 工具调用fromsim.tf_idfimportTFIdftokens_list=["这是 一个 什么 样 的 工具","..."]query=["非常 好用 的 工具"]tf_idf=TFIdf(tokens_list,split=" ")print(tf_idf.get_score(query,0))# scoreprint(tf_idf.get_score_list(query,10))# [(index, score), ...]print(tf_idf.weight())# list or numpy arrayBM25# Examplefromexamples.run_bm25importactuatoractuator("./corpus/chinese/breeno/train.tsv",query1="12 23 4160 276",query2="29 23 169 1495")# 工具调用fromsim.bm25importBM25tokens_list=["这是 一个 什么 样 的 工具","..."]query=["非常 好用 的 工具"]bm25=BM25(tokens_list,split=" ")print(bm25.get_score(query,0))# scoreprint(bm25.get_score_list(query,10))# [(index, score), ...]print(bm25.weight())# list or numpy arrayLSHfromsim.lshimportE2LSHfromsim.lshimportMinHashe2lsh=E2LSH()min_hash=MinHash()candidates=[[3.6216,8.6661,-2.8073,-0.44699,0],...]query=[-2.7769,-5.6967,5.9179,0.37671,1]print(e2lsh.search(candidates,query))# index in candidatesprint(min_hash.search(candidates,query))# index in candidatesSIFA Simple But Tough-To-Beat Baseline For Sentence EmbeddingsUnsupervised Random Walk Sentence Embeddings: A Strong but Simple Baselinesentences=[["token1","token2","..."],...]vector=[[[1,1,1],[2,2,2],[...]],...]fromsim.sif_usifimportSIFfromsim.sif_usifimportuSIFsif=SIF(n_components=5,component_type="svd")sif.fit(tokens_list=sentences,vector_list=vector)usif=uSIF(n_components=5,n=1,component_type="svd")usif.fit(tokens_list=sentences,vector_list=vector)FastTextBag of Tricks for Efficient Text Classification# TensorFlow versionfromexamples.tensorflow.run_fast_textimportactuatoractuator(execute_type="train",model_type="bert",model_dir="./data/chinese_wwm_L-12_H-768_A-12")# Pytorch versionfromexamples.pytorch.run_fast_textimportactuatoractuator(execute_type="train",model_type="bert",model_dir="./data/chinese_wwm_pytorch")RNN BaseSiamese Recurrent Architectures for Learning Sentence SimilarityLearning Text Similarity with Siamese Recurrent Networks# TensorFlow versionfromexamples.tensorflow.run_siamese_rnnimportactuatoractuator("./data/config/siamse_rnn.json",execute_type="train")# Pytorch versionfromexamples.pytorch.run_siamese_rnnimportactuatoractuator("./data/config/siamse_rnn.json",execute_type="train")CNN BaseConvolutional Neural Networks for Sentence ClassificationCharacter-Aware Neural Language ModelsHighway NetworksVery Deep Convolutional Networks for Text Classification# TensorFlow versionfromexamples.tensorflow.run_cnn_baseimportactuatoractuator(execute_type="train",model_type="bert",model_dir="./data/chinese_wwm_L-12_H-768_A-12")# Pytorch versionfromexamples.pytorch.run_cnn_baseimportactuatoractuator(execute_type="train",model_type="bert",model_dir="./data/chinese_wwm_pytorch")Bert BaseAttention Is All You Need# TensorFlow versionfromexamples.tensorflow.run_basic_bertimportactuatoractuator(model_dir="./data/chinese_wwm_L-12_H-768_A-12",execute_type="train")# Pytorch versionfromexamples.pytorch.run_basic_bertimportactuatoractuator(model_dir="./data/chinese_wwm_pytorch",execute_type="train")AlbertALBERT: A Lite BERT For Self-superpised Learning Of Language Representations# TensorFlow versionfromexamples.tensorflow.run_albertimportactuatoractuator(model_dir="./data/albert_small_zh_google",execute_type="train")# Pytorch versionfromexamples.pytorch.run_albertimportactuatoractuator(model_dir="./data/albert_chinese_small",execute_type="train")NEZHANEZHA: Neural Contextualized Representation For Chinese Language Understanding# TensorFlow versionfromexamples.tensorflow.run_nezhaimportactuatoractuator(model_dir="./data/NEZHA-Base-WWM",execute_type="train")# Pytorch versionfromexamples.pytorch.run_nezhaimportactuatoractuator(model_dir="./data/nezha-base-wwm",execute_type="train")RoBERTaRoBERTa: A Robustly Optimized BERT Pretraining Approach# TensorFlow versionfromexamples.tensorflow.run_basic_bertimportactuatoractuator(model_dir="./data/chinese_roberta_L-6_H-384_A-12",execute_type="train")# Pytorch versionfromexamples.pytorch.run_basic_bertimportactuatoractuator(model_dir="./data/chinese-roberta-wwm-ext",execute_type="train")SimCSESimCSE: Simple Contrastive Learning of Sentence Embeddings# TensorFlow versionfromexamples.tensorflow.run_simcseimportactuatoractuator(model_dir="./data/chinese_wwm_L-12_H-768_A-12",execute_type="train",model_type="bert")# Pytorch versionfromexamples.pytorch.run_simcseimportactuatoractuator(model_dir="./data/chinese_wwm_pytorch",execute_type="train",model_type="bert")Poly-EncoderPoly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring# TensorFlow versionfromexamples.tensorflow.run_poly_encoderimportactuatoractuator(model_dir="./data/chinese_wwm_L-12_H-768_A-12",execute_type="train",model_type="bert")# Pytorch versionfromexamples.pytorch.run_poly_encoderimportactuatoractuator(model_dir="./data/chinese_wwm_pytorch",execute_type="train",model_type="bert")ColBERTColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT# TensorFlow versionfromexamples.tensorflow.run_colbertimportactuatoractuator(model_dir="./data/chinese_wwm_L-12_H-768_A-12",execute_type="train",model_type="bert")# Pytorch versionfromexamples.pytorch.run_colbertimportactuatoractuator(model_dir="./data/chinese_wwm_pytorch",execute_type="train",model_type="bert")RE2Simple and Effective Text Matching with Richer Alignment Features# TensorFlow versionfromexamples.tensorflow.run_re2importactuatoractuator("./data/config/re2.json",execute_type="train")# Pytorch versionfromexamples.pytorch.run_re2importactuatoractuator("./data/config/re2.json",execute_type="train")Cite@misc{text-similarity,
  title={text-similarity},
  author={Bocong Deng},
  year={2021},
  howpublished={\url{https://github.com/DengBoCong/text-similarity}},
}Referencebert4kerasalbert_zhHuggingFaceSelf-Attention with Relative Position Representations