Repository: flairNLP/flair

Issue:
Tokenization MISMATCH causes Runtime Error (keyphrase tagger model)
While training a tagging model using the provided Keyphrase dataset SEMEVAL2017, following error comes up:

Tokenization MISMATCH in sentence '...'
Last matched: 'Token: 272 ̄'
Last sentence: 'Token: 325 .'

which causes directly afterwards a Runtime Error during forward method in sequence_tagger_model.py:
File "xxx/PycharmProjects/flair/flair/models/sequence_tagger_model.py", line 541, in forward
    self.embeddings.embedding_length,
RuntimeError: shape '[16, 326, 3072]' is invalid for input of size 15857664

To reproduce, use a tagger tutorial code from flair. I use TransformerWordEmbeddings() with scibert or bert-base-uncased as embedding_types. Error does not comes up for example when training the tagger with flair embeddings.

Expected is that the trainer can complete training without stopping due to special characters in the sentences.

Environment:
 - MacOS Catalina 10.15.4
 - flair version 0.5



Discussion:
A list of sentences doesn't working:

- The spins and parities of Θ + and Ξ −− are not yet known experimentally . In this new wave of pentaquark research , most theoretical papers take the spin equal to 1 / 2 . The parity is more controversial . In chiral soliton or Skyrme models the parity is positive [ 4 ] . In constituent quark models it is usually positive . In the present approach , the parity of the pentaquark is given by P =(−) ℓ + 1 , where ℓ is the angular momentum associated with the relative coordinates of the q4 subsystem . We analyze the case where the subsystem of four light quarks is in a state of orbital symmetry [ 31 ] O and carries an angular momentum ℓ = 1 . Although the kinetic energy of such a state is higher than that of the totally symmetric [ 4 ] O state , the [ 31 ] O symmetry is the most favourable both for the flavour – spin interaction [ 12 ] and the colour – spin interaction [ 13 ] . In the first case the statement is confirmed by the comparison between the realistic calculations for positive parity [ 12 ] and negative parity [ 14 ] , based on the same quark model [ 15 ] . In Ref . [ 12 ] the antiquark was heavy , c or b , and accordingly the interaction between light quarks and the heavy antiquark was neglected , consistent with the heavy quark limit . In Ref . [ 16 ] an attractive spin – spin interaction between s ̄ and the light quarks was incorporated and shown that a stable or narrow positive parity uudds ̄ pentaquark can be accommodated within such a model . This interaction has a form that corresponds to η meson exchange [ 17 ] and its role is to lower the energy of the whole system .
- Absorption events through the charged current reactions ( 2 ) νe + 40Ar → e −+ 40K ∗ andν ̄ e + 40Ar → e ++ 40Cl ∗ . There is some uncertainty in predicting e −( e +) event rates for these processes which arise due to the nuclear model dependencies of the absorption cross section and the treatment of the Coulomb distortion of electron ( positron ) in the field of the residual nucleus . The nuclear absorption cross section for the charged current neutrino reactions in 40Ar relevant to supernova neutrino energies was first calculated by Raghavan [ 10 ] and Bahcall et al . [ 11 ] for Fermi transitions leading to isobaric analogue state ( IAS ) at 4.38 MeV in 40K ∗ . Later Ormand et al . [ 12 ] used a shell model to calculate the Fermi and Gamow – Teller transitions . In these calculations Fermi function F ( Z,Ee ) was used to take into account the Coulomb effects . In a recent paper Bueno et al . [ 13 ] make use of a calculation by Martinez-Pinedo et al . [ 14 ] who use a shell model for Fermi and Gamow – Teller transitions and a continuum random phase approximation ( CRPA ) for forbidden transitions to calculate the absorption cross sections . In this calculation the Coulomb distortion of the produced electron is treated with a hybrid model where a Fermi function is used for lower electron energies and modified effective momentum approximation ( MEMA ) for higher electron energies [ 14 – 17 ] . In a recent work Bhattacharya et al . [ 18 ] have measured the Fermi and Gamow – Teller transition strengths leading to excited states up to 6 MeV in 40K ∗ and obtained the neutrino absorption cross section for supernova neutrinos in 40Ar .
- States outside the constituent quark model have been hypothesized to exist almost since the introduction of color [ 1 – 4 ] . Hybrid mesons , qq ̄ states with an admixture of gluons , and glueballs , states with no quark content , rely on the self interaction property of gluons due to their color charge . Looking for glueballs would be the most obvious way to find evidence for states with constituent gluons ; however , the search is hindered by the fact that these states may significantly mix with regular qq ̄- mesons in the region where the lightest are predicted to occur . As such , they may not be observable as pure states and disentangling the observed spectra may be a very difficult task . Instead , hybrid mesons ( qq ̄ gn ) may be a better place to search for evidence of resonances outside the constituent quark model , especially since the lightest of theses states are predicted to have exotic quantum numbers of spin , parity , and charge conjugation , JPC , that is , combinations that are unattainable by regular qq ̄- mesons .
- In contrast to the H particle , the situation for the Θ + baryon is very promising . Thus , in this Letter we explore the formation of the Θ +- baryon within a new approach called parton-based Gribov – Regge theory . It is realized in the Monte Carlo program NEXUS 3.97 [ 22,23 ] . In this model high energy hadronic and nuclear collisions are treated within a self-consistent quantum mechanical multiple scattering formalism . Elementary interactions , happening in parallel , correspond to underlying microscopic ( predominantly soft ) parton cascades and are described effectively as phenomenological soft pomeron exchanges . A pomeron can be seen as layers of a ( soft ) parton ladder , which is attached to projectile and target nucleons via leg partons . At high energies one accounts also for the contribution of perturbative ( high pt ) partons described by a so-called “ semihard pomeron ”— a piece of the QCD parton ladder sandwiched between two soft pomerons which are connected to the projectile and to the target in the usual way . The spectator partons of both projectile and target nucleons , left after pomeron emissions , form nucleon remnants . The legs of the pomerons form color singlets , such as q – q ̄ , q – qq or q ̄– q ̄ q ̄ . The probability of q – qq and q ̄– q ̄ q ̄ is controlled by the parameter Pqq and is fixed by the experimental yields on ( multi -) strange baryons [ 23 ] .
- Thus , the extension to the charmed analogue Θc ( 3099 ) provides an interesting test for the SDO sum rule and lattice calculations [ 17 ] . Here , the charm quark is quite heavy so that the constituent-quark picture may fit well and the JW prediction for the parity is expected to be reproduced from QCD . In fact , quenched lattice calculation finds the parity of Θc ( 3099 ) to be positive [ 28 ] . In the extension to the Θc ( 3099 ) sum rules , there are two important aspects , which make this sum rule different from the SDO sum rule . First of all , since the charm quark is too heavy to form quark condensate , it gives non-perturbative effects only by radiating gluons . The quark – gluon mixed condensate 〈 s ̄ gsσ · Gs 〉 , which was the important contribution in the Θ + sum rule , is replaced by gluonic operators in the heavy quark expansion that are normally suppressed . Secondly , the charm quark mass has to be kept finite in the OPE , which can be done by using the momentum space expression for the charm-quark propagator . This is different from the light-quark sum rule where the calculation is performed in the coordinate space and all the quark propagators are obtained based on the expansion with the small quark mass . Keeping these two aspects in mind , we construct QCD sum rules for Θc ( 3099 ) and see how they are different from the Θ +( 1540 ) sum rule .
- Some non-standard couplings , which should be determined here , could also be studied in the standard e + e − option of a linear collider . Therefore , it is worth while to compare the potential power of the two options . As far as the parameter αγ1 is concerned , the γγ collider does not allow for its determination , while it could be determined at e + e − . The second tt ̄ γ coupling αγ2 , which is proportional to the real part of the top-quark electric dipole moment,44See [ 23 ] taking into account that the operators OuB , OqB and OqW are redundant . can be measured here . It should be recalled that energy and polar-angle distributions of leptons and b-quarks in e + e − colliders are sensitive only to the imaginary part of the electric dipole moment,55However , it should be emphasized that there exist observables sensitive also to the real part of the top-quark electric dipole moment , see [ 24 ] . while here the real part could be determined . For the measurement of γγH couplings , e + e − colliders are , of course , useless , while here , for the bX final state both αh1 and αh2 could be measured . In the case of the decay form factor αd measurement , the e + e − option seems to be a little more advantageous , especially if e + e − polarization can be tuned appropriately [ 25 ] .
- One major goal of current nuclear physics is the observation of at least partial restoration of chiral symmetry . Since the chiral order parameter 〈 q ̄ q 〉 is expected to decrease by about 30 % already at normal nuclear matter density [ 1 – 4 ] , any in-medium change due to the dropping quark condensate should in principle be observable in photonuclear reactions . The conjecture that such a partial restoration of chiral symmetry causes a softening and narrowing of the σ meson as the chiral partner of the pion in the nuclear medium [ 5,6 ] has led to the idea of measuring the π0π0 invariant mass distribution near the 2π threshold in photon induced reactions on nuclei [ 7 ] . In contrast to its questionable nature as a proper quasiparticle in vacuum , the σ meson might develop a much narrower peak at finite baryon density due to phase-space suppression for the σ → ππ decay , hence making it possible to explore its properties when embedded in a nuclear many-body system [ 8 – 11 ] . Measuring a threshold enhancement of the π0π0 invariant mass spectrum might serve as a signal for the partial restoration of chiral symmetry inside nuclei and , therefore , give information about one of the most fundamental features of QCD .
- Production of charmonium states J / ψ and ψ′ in nucleus – nucleus collisions has been studied at CERN SPS over the previous 15 years by the NA38 and NA50 Collaborations . This experimental program was mainly motivated by the suggestion [ 1 ] to use the J / ψ as a probe of the state of matter created at the early stage of the collision . The original picture [ 1 ] ( see also [ 2 ] for a modern review ) assumes that charmonia are created exclusively at the initial stage of the reaction in primary nucleon – nucleon collisions . During the subsequent evolution of the system , the number of hidden charm mesons is reduced because of : ( a ) absorption of pre-resonance charmonium states by nuclear nucleons ( normal nuclear suppression ) , ( b ) interactions of charmonia with secondary hadrons ( comovers ) , ( c ) dissociation of cc ̄ bound states in deconfined medium ( anomalous suppression ) . It was found [ 3 ] that J / ψ suppression with respect to Drell – Yan muon pairs measured in proton – nucleus and nucleus – nucleus collisions with light projectiles can be explained by the so-called “ normal ” ( due to sweeping nucleons ) nuclear suppression alone . In contrast , the NA50 experiment with a heavy projectile and target ( Pb + Pb ) revealed essentially stronger J / ψ suppression for central collisions [ 4 – 7 ] . This anomalous J / ψ suppression was attributed to formation of quark – gluon plasma ( QGP ) [ 7 ] , but a comover scenario cannot be excluded [ 8 ] .

Resolving patch:
diff --git a/flair/embeddings/token.py b/flair/embeddings/token.py
--- a/flair/embeddings/token.py
+++ b/flair/embeddings/token.py
@@ -921,7 +921,6 @@ def _add_embeddings_to_sentences(self, sentences: List[Sentence]):
 
             subtokenized_sentences.append(torch.tensor(subtokenized_sentence, dtype=torch.long))
             subtokens = self.tokenizer.convert_ids_to_tokens(subtokenized_sentence)
-            # print(subtokens)
 
             word_iterator = iter(sentence)
             token = next(word_iterator)
@@ -934,19 +933,24 @@ def _add_embeddings_to_sentences(self, sentences: List[Sentence]):
             # iterate over subtokens and reconstruct tokens
             for subtoken_id, subtoken in enumerate(subtokens):
 
-                subtoken_count += 1
-
                 # remove special markup
                 subtoken = self._remove_special_markup(subtoken)
 
+                # check if subtoken is special begin token ([CLS] or similar)
+                if subtoken in self.special_tokens and subtoken_id == 0:
+                    continue
+
+                # some BERT tokenizers somehow omit words - in such cases skip to next token
+                if subtoken_count == 0 and not token_text.startswith(subtoken.lower()):
+                    token_subtoken_lengths.append(0)
+                    token = next(word_iterator)
+                    token_text = self._get_processed_token_text(token)
+
+                subtoken_count += 1
+
                 # append subtoken to reconstruct token
                 reconstructed_token = reconstructed_token + subtoken
 
-                # check if reconstructed token is special begin token ([CLS] or similar)
-                if reconstructed_token in self.special_tokens and subtoken_id == 0:
-                    reconstructed_token = ''
-                    subtoken_count = 0
-
                 # check if reconstructed token is the same as current token
                 if reconstructed_token.lower() == token_text:
 
@@ -957,10 +961,11 @@ def _add_embeddings_to_sentences(self, sentences: List[Sentence]):
                     reconstructed_token = ''
                     subtoken_count = 0
 
-                    # break from loop if all tokens are accounted for
+                    # get next token
                     if len(token_subtoken_lengths) < len(sentence):
                         token = next(word_iterator)
                         token_text = self._get_processed_token_text(token)
+                    # break from loop if all tokens are accounted for
                     else:
                         break
 
@@ -1008,6 +1013,11 @@ def _add_embeddings_to_sentences(self, sentences: List[Sentence]):
                 # for each token, get embedding
                 for token_idx, (token, number_of_subtokens) in enumerate(zip(sentence, subtoken_lengths)):
 
+                    # some tokens have no subtokens at all (if omitted by BERT tokenizer) so return zero vector
+                    if number_of_subtokens == 0:
+                        token.set_embedding(self.name, torch.zeros(self.embedding_length))
+                        continue
+
                     subword_end_idx = subword_start_idx + number_of_subtokens
 
                     subtoken_embeddings: List[torch.FloatTensor] = []


Test patch:
diff --git a/tests/test_embeddings.py b/tests/test_embeddings.py
--- a/tests/test_embeddings.py
+++ b/tests/test_embeddings.py
@@ -103,6 +103,21 @@ def test_transformer_word_embeddings():
     del embeddings
 
 
+def test_transformer_weird_sentences():
+
+    embeddings = TransformerWordEmbeddings('distilbert-base-uncased', layers='all', use_scalar_mix=True)
+
+    sentence = Sentence("Hybrid mesons , qq ̄ states with an admixture")
+    embeddings.embed(sentence)
+    for token in sentence:
+        assert len(token.get_embedding()) == 768
+
+    sentence = Sentence("typical proportionalities of ∼ 1nmV − 1 [ 3,4 ] .")
+    embeddings.embed(sentence)
+    for token in sentence:
+        assert len(token.get_embedding()) == 768
+
+
 def test_fine_tunable_flair_embedding():
     language_model_forward = LanguageModel(
         Dictionary.load("chars"), is_forward_lm=True, hidden_size=32, nlayers=1