# Compositional Distributional Semantics with Syntactic Dependencies and Sectional Preferences
## 3 A Dependency-Based Compositional Strategy for Meaning Construction
### System's Configuration
#### 4.2.2 BERT-Like Transformers  
Concerning the Transformers, for English, we made use of Sentence-BERT (SBERT), a modification of the pre-trained BERT-Large (with 24 layers), to derive fixed sized vectors of sentences [31]. It is very important to note that SBERT for English was fine-tuned with two very large dataset collections, both SNLI [32] and MultiNLI [33] containing 1 million sentence pairs which were annotated for semantic tasks such as inference, contradiction and entailment. In addition, to generate the dynamic vectors of words in context, we used BERT-Large model by adding the 4 last layers of each word. These specific configurations turned out to give the best performances in the experiments reported in [29].  
For Spanish and Portuguese, we used BETO [34], with a size similar to BERT-Base (12 layers), and BERTimbau Large [35], with 24 layers, respectively. They are both accessible from HuggingFace Models Hub ([[https://huggingface.co/dccuchile/bert-base-spanish-wwm-cased](https://huggingface.co/dccuchile/bert-base-spanish-wwm-cased)]([https://huggingface.co/dccuchile/bert-base-spanish-wwm-cased](https://huggingface.co/dccuchile/bert-base-spanish-wwm-cased)) (accessed on 15 June 2021) and [[https://huggingface.co/neuralmind/bert-large-portuguese-cased](https://huggingface.co/neuralmind/bert-large-portuguese-cased)]([https://huggingface.co/neuralmind/bert-large-portuguese-cased](https://huggingface.co/neuralmind/bert-large-portuguese-cased)) (accessed on 15 June 2021)). To derive fixed sized sentence embedding, we compute the mean of all output vectors. This is the same strategy as that used by SBERT for English. The main difference with regard to SBERT is that the pre-trained models are not fine-tuned with annotated collections of semantically similar pairs of sentences. As previously shown for English, contextualized vectors were generated by adding the 4 last layers of each word in context.  
In sum, four different BERT configurations were used in the experiments:  
**SBERT-sentence**: Sentence vector derived from the mean of all output vectors and fine-tuned on NLI and STS datasets.
**bert-sentence**: Sentence vector derived in the same way but without fine-tuning.
**bert-head**: Contextualized vector of the verb head (adding 4 last layers)
**bert-dep-subj**: Contextualized vector of the subject word (adding 4 last layers).
**bert-dep-obj**: Contextualized vector of the direct object (adding 4 last layers).  
We consider that any of the three words of the noun-verb-noun expression can semantically represents the whole sentence, as all constituent words are fully contextualized in a BERT-like Transformer architecture.  
All systems were evaluated against three datasets (one per language) with examples codified in subject-verb-object triples. However, as we have said before, in order to provide a more appropriate scenario for Transformers, lemma triples were converted into full grammatical sentences. So, for each BERT configuration we defined two different tests:  
**Lemma triple**: this corresponds to the original subject-verb-object triple.
**Full sentence**: each triple was transformed into a regular inflected sentence by using the following procedure: by default, a definite determiner is added to the subject, the verb is written in the present tense and the object is preceded by an indefinite determiner. The cases in which this by default procedure does not work have been revised and corrected.