Package: crosslingual-coreference
Crosslingual CoreferenceCoreference is amazing but the data required for training a model is very scarce. In our case, the available training for non-English languages also proved to be poorly annotated. Crosslingual Coreference, therefore, uses the assumption a trained model with English data and cross-lingual embeddings should work for languages with similar sentence structures.Installpip install crosslingual-coreferenceQuickstartfromcrosslingual_coreferenceimportPredictortext=("Do not forget about Momofuku Ando! He created instant noodles in Osaka. At"" that location, Nissin was founded. Many students survived by eating these"" noodles, but they don't even know him.")# choose minilm for speed/memory and info_xlm for accuracypredictor=Predictor(language="en_core_web_sm",device=-1,model_name="minilm")print(predictor.predict(text)["resolved_text"])print(predictor.pipe([text])[0]["resolved_text"])# Note you can also get 'cluster_heads' and 'clusters'# Output## Do not forget about Momofuku Ando!# Momofuku Ando created instant noodles in Osaka.# At Osaka, Nissin was founded.# Many students survived by eating instant noodles,# but Many students don't even know Momofuku Ando.ModelsAs of now, there are two models available "spanbert", "info_xlm", "xlm_roberta", "minilm", which scored 83, 77, 74 and 74 on OntoNotes Release 5.0 English data, respectively.The "minilm" model is the best quality speed trade-off for both mult-lingual and english texts.The "info_xlm" model produces the best quality for multi-lingual texts.The AllenNLP "spanbert" model produces the best quality for english texts.Chunking/batching to resolve memory OOM errorsfromcrosslingual_coreferenceimportPredictorpredictor=Predictor(language="en_core_web_sm",device=0,model_name="minilm",chunk_size=2500,chunk_overlap=2,)Use spaCy pipelineimportspacytext=("Do not forget about Momofuku Ando! He created instant noodles in Osaka. At"" that location, Nissin was founded. Many students survived by eating these"" noodles, but they don't even know him.")nlp=spacy.load("en_core_web_sm")nlp.add_pipe("xx_coref",config={"chunk_size":2500,"chunk_overlap":2,"device":0})doc=nlp(text)print(doc._.coref_clusters)# Output## [[[4, 5], [7, 7], [27, 27], [36, 36]],# [[12, 12], [15, 16]],# [[9, 10], [27, 28]],# [[22, 23], [31, 31]]]print(doc._.resolved_text)# Output## Do not forget about Momofuku Ando!# Momofuku Ando created instant noodles in Osaka.# At Osaka, Nissin was founded.# Many students survived by eating instant noodles,# but Many students don't even know Momofuku Ando.print(doc._.cluster_heads)# Output## {Momofuku Ando: [5, 6],# instant noodles: [11, 12],# Osaka: [14, 14],# Nissin: [21, 21],# Many students: [26, 27]}Visualize spacy pipelineThis only works with spacy >= 3.3.importspacyfromspacy.tokensimportSpanfromspacyimportdisplacytext=("Do not forget about Momofuku Ando! He created instant noodles in Osaka. At"" that location, Nissin was founded. Many students survived by eating these"" noodles, but they don't even know him.")nlp=spacy.load("nl_core_news_sm")nlp.add_pipe("xx_coref",config={"model_name":"minilm"})doc=nlp(text)spans=[]foridx,clusterinenumerate(doc._.coref_clusters):forspanincluster:spans.append(Span(doc,span[0],span[1]+1,str(idx).upper()))doc.spans["custom"]=spansdisplacy.render(doc,style="span",options={"spans_key":"custom"})More Examples