Projeto commited on
Commit
a340739
1 Parent(s): 00a6cc8

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +478 -0
README.md ADDED
@@ -0,0 +1,478 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # ***LegalNLP*** - Natural Language Processing Methods for the Brazilian Legal Language :balance_scale:
2
+
3
+ ### The library of Natural Language Processing for Brazilian legal language, *LegalNLP*, was born in a partnership between Brazilian researchers and the legal tech [Tikal Tech](https://www.tikal.tech) based in São Paulo, Brazil. Besides containing pre-trained language models for the Brazilian legal language, ***LegalNLP*** provides functions that can facilitate the manipulation of legal texts in Portuguese and demonstration/tutorials to help people in their own work.
4
+
5
+ You can access our paper by clicking [**here**](https://arxiv.org/abs/2110.15709).
6
+
7
+ If you use our library in your academic work, please cite us in the following way
8
+
9
+ @article{polo2021legalnlp,
10
+ title={LegalNLP--Natural Language Processing methods for the Brazilian Legal Language},
11
+ author={Polo, Felipe Maia and Mendon{\c{c}}a, Gabriel Caiaffa Floriano and Parreira, Kau{\^e} Capellato J and Gianvechio, Lucka and Cordeiro, Peterson and Ferreira, Jonathan Batista and de Lima, Leticia Maria Paz and Maia, Ant{\^o}nio Carlos do Amaral and Vicente, Renato},
12
+ journal={arXiv preprint arXiv:2110.15709},
13
+ year={2021}
14
+ }
15
+
16
+ --------------
17
+
18
+ ## Summary
19
+
20
+ 0. [Accessing the Language Models](#0)
21
+ 1. [ Introduction / Installing package](#1)
22
+ 2. [Fuctions ](#2)
23
+ 1. [ Text Cleaning Functions](#2.1)
24
+ 2. [Other Functions](#2.2)
25
+ 3. [ Language Models (Details / How to use)](#3)
26
+ 1. [ Phraser ](#3.1)
27
+ 2. [ Word2Vec/Doc2Vec ](#3.2)
28
+ 3. [ FastText ](#3.3)
29
+ 4. [ BERTikal ](#3.4)
30
+ 4. [ Demonstrations / Tutorials](#4)
31
+ 5. [ References](#5)
32
+
33
+ --------------
34
+
35
+ <a name="0"></a>
36
+ ## 0\. Accessing the Language Models
37
+
38
+
39
+ All our models can be found [here](https://drive.google.com/drive/folders/1tCccOXPLSEAEUQtcWXvED3YaNJi3p7la?usp=sharing).
40
+
41
+ Some models can be download directly using our function `get_premodel` (more details in section [Other Functions](#2.2)).
42
+
43
+
44
+ Please contact *felipemaiapolo@gmail.com* if you have any problem accessing the language models.
45
+
46
+ --------------
47
+
48
+ <a name="1"></a>
49
+ ## 1\. Introduction / Installing package
50
+ *LegalNLP* is promising given the scarcity of Natural Language Processing resources focused on the Brazilian legal language. It is worth mentioning that our library was made for Python, one of the most well-known programming languages for machine learning.
51
+
52
+
53
+ You can install our package running the following command on terminal
54
+ ``` :sh
55
+ $ pip install git+https://github.com/felipemaiapolo/legalnlp
56
+ ```
57
+
58
+ You can load all our functions running the following command
59
+
60
+ ```python
61
+ from legalnlp.clean_functions import *
62
+ from legalnlp.get_premodel import *
63
+ ```
64
+
65
+
66
+ --------------
67
+
68
+ <a name="2"></a>
69
+ ## 2\. Functions
70
+ <a name="2.1"></a>
71
+ ### 2.1\. Text Cleaning Functions
72
+
73
+
74
+ <a name="2.1.1"></a>
75
+ #### 2.1.1\. `clean(text, lower=True, return_masked=False)`
76
+ Function for cleaning texts to be used (optional) in conjunction with Doc2Vec, Word2Vec, and FastText models. We use RegEx to mask/extract information such as email addresses, URLs, dates, numbers, monetary values, etc.
77
+
78
+ **input**:
79
+
80
+ - *text*, **str**;
81
+
82
+ - *lower*, **bool**, default=**True**. If lower==True, function lower cases the whole text. Note that all the models (except BERT) were trained with lower cased texts;
83
+
84
+ - *return_masked*, **bool**, default=**True**. If return_masked == False, the function outputs a clean text. Otherwise, it returns a dictionary containing the clean text and the information extracted by RegEx;
85
+
86
+ **output**:
87
+
88
+ - Clean text or dictionary, depending on the *return_masked* parameter;
89
+
90
+
91
+ <a name="2.1.2"></a>
92
+ #### 2.1.2\.`clean_bert(text)`
93
+
94
+ Function for cleaning the texts to be used (optional) in conjunction with the BERT model.
95
+
96
+ **input:**
97
+
98
+ - *text*, **str**.
99
+
100
+ **output:**
101
+
102
+ - **str** with clean text.
103
+
104
+ <a name="2.2"></a>
105
+ ### 2.2\. Other functions
106
+
107
+ #### 2.2.2\. `get_premodel(model)`
108
+
109
+ Function to download a pre-trained model in the same folder as the file that is being executed.
110
+
111
+ **input:**
112
+
113
+ - *model*, **str**. Must contain the name of the pre-trained model that one wants to use. There are these options:
114
+ - **model = "bert"**: Download a .zip file containing BERTikal model and unzip it.
115
+ - **model = "wdoc"**: Download Word2Vec and Do2vec pre-trained models in a.zip file and unzip it. It has 2 two files, one with an size 100 Doc2Vec Distributed Memory/ Word2Vec Continuous Bag-of-Words (CBOW) embeddings model and other with an size 100 Doc2Vec Distributed Bag-of-Words (DBOW)/ Word2Vec Skip-Gram (SG) embeddings model.
116
+ - **model = "fasttext"**: Download a .zip file containing 100 sized FastText CBOW/SG models and unzip it.
117
+ - **model = "phraser"**: Download Phraser pre-trained model in a .zip file and unzip it. It has 2 two files with phraser1 and phreaser2. We explain how to use them in Section [ Phraser ](#3.1).
118
+ - **model = "w2vnilc"**: Download size 100 Word2Vec CBOW model trained by "Núcleo Interinstitucional de Linguística Computacional - USP" embeddings model in a .zip file and unzip it. [Click here for more details](http://nilc.icmc.usp.br/nilc/index.php/repositorio-de-word-embeddings-do-nilc).
119
+ - **model = "neuralmindbase"**: Download a .zip file containing base BERT model (PyTorch), trained by NeuralMind and unzip it. For more informations about BERT models made by NeuralMind go to [their GitHub repo](https://github.com/neuralmind-ai/portuguese-bert).
120
+ - **model = "neuralmindlarge"**: Download a .zip file containing large BERT model (PyTorch), trained by NeuralMind and unzip it. For more informations about BERT models made by NeuralMind go to [their GitHub repo](https://github.com/neuralmind-ai/portuguese-bert).
121
+
122
+
123
+ **output:**
124
+
125
+ - True if download of some model was made and False otherwise.
126
+
127
+
128
+ #### 2.2.1\. `extract_features_bert(path_model, path_tokenizer, data, gpu=True)`
129
+
130
+ Function for extracting features with the BERT model (This function is not accessed through the package installation, but you can find it [here](https://github.com/felipemaiapolo/legalnlp/blob/main/demo/BERT/extract_features_bert.ipynb)).
131
+
132
+
133
+ **Input:**
134
+
135
+ - *path_model*, **str**. Must contain the path of the pre-trained model;
136
+
137
+ - *path_tokenizer*, **str**. Must contain the path of tokenizer;
138
+
139
+ - *data*, **list**. Must contain a list of texts that will be extracted features;
140
+
141
+ - *gpu*, **bool**, default=**True**. If gpu==False, the GPU will not be used in the model application (we recommend feature extraction to be done using Google Colab).
142
+
143
+
144
+ **Output:**
145
+
146
+ - **DataFrame** with features extracted by BERT model.
147
+
148
+
149
+ <a name="3"></a>
150
+ ## 3\. Model Languages
151
+
152
+ <a name="3.1"></a>
153
+ ### 3.1\. Phraser
154
+
155
+ Phraser is a statistical method proposed in the natural language processing
156
+ literature [1] for identifying which words when they appear
157
+ together, can be considered as unique tokens. This method application is able to
158
+ identify the relevance of the occurrence of a bigram against the occurrence of the
159
+ words that make it up separately. Thus, we can identify that a bigram like "São
160
+ Paulo" should be treated as a single token, for example. If the method is applied
161
+ a second time in sequence, we can check which are the relevant trigrams and
162
+ quadrigrams. Since the two applications should be done with different Phraser
163
+ models, it can be the case that the second application identifies bigrams that were
164
+ not identified by the first model.
165
+
166
+ This model is compatible with the `clean` function, but it is not necessary to use it before. Remember to at least make all letters lowercase. Please check our paper or [Gensim page](https://radimrehurek.com/gensim_3.8.3/models/phrases.html) for more details. Preferably use Gensim version 3.8.3.
167
+
168
+ #### Using *Phraser*
169
+ Installing Gensim
170
+
171
+
172
+ ```python
173
+ !pip install gensim=='3.8.3'
174
+ ```
175
+
176
+ Importing package and loading our two Phraser models.
177
+
178
+
179
+ ```python
180
+ #Importing packages
181
+ from gensim.models.phrases import Phraser
182
+
183
+ #Loading two Phraser models
184
+ phraser1=Phraser.load('models_phraser/phraser1')
185
+ phraser2=Phraser.load('models_phraser/phraser2')
186
+ ```
187
+
188
+
189
+ Applying Phraser once and twice to check output
190
+
191
+
192
+ ```python
193
+ txt='direito do consumidor origem : bangu regional xxix juizado especial civel ação : [processo] - - recte : fundo de investimento em direitos creditórios'
194
+ tokens=txt.split()
195
+
196
+ print('Clean Text: "'+' '.join(tokens)+'"')
197
+ print('\nApplying Phraser 1x: "'+' '.join(phraser1[tokens])+'"')
198
+ print('\nApplying Phraser 2x: "'+' '.join(phraser2[phraser1[tokens]])+'"')
199
+ ```
200
+
201
+ Clean Text: "direito do consumidor origem : bangu regional xxix juizado especial civel ação : [processo] - - recte : fundo de investimento em direitos creditórios"
202
+
203
+ Applying Phraser 1x: "direito do consumidor origem : bangu regional xxix juizado_especial civel_ação : [processo] - - recte : fundo de investimento em direitos_creditórios"
204
+
205
+ Applying Phraser 2x: "direito do consumidor origem : bangu_regional xxix juizado_especial_civel_ação : [processo] - - recte : fundo de investimento em direitos_creditórios"
206
+
207
+ <a name="3.2"></a>
208
+ ### 3.2\. Word2Vec/Doc2Vec
209
+
210
+ Our first models for generating vector representation for tokens and
211
+ texts (embeddings) are variations of the Word2Vec [1,
212
+ 2] and Doc2Vec [3] methods. In short, the
213
+ Word2Vec methods generate embeddings for tokens5 and that somehow capture
214
+ the meaning of the various textual elements, based on the contexts in which these
215
+ elements appear. Doc2Vec methods are extensions/modifications of Word2Vec
216
+ for generating whole text representations.
217
+
218
+ The Word2Vec and Doc2Vec methods are presented together in this section because they were trained together using the Gensim package. Both models are compatible with the `clean` function, but it is not necessary to use it before. Remember to at least make all letters lowercase. Please check our paper or [Gensim page](https://radimrehurek.com/gensim_3.8.3/models/doc2vec.html) for more details. Preferably use Gensim version 3.8.3.
219
+
220
+
221
+ Below we have a summary table with some important information about the trained models:
222
+
223
+
224
+
225
+ | Filenames | Doc2Vec | Word2Vec | Size | Windows
226
+ |:-------------------:|:--------------:|:--------------:|:--------------:|:--------------:|
227
+ | ```w2v_d2v_dm*``` | Distributed Memory (DM) | Continuous Bag-of-Words (CBOW) | 100, 200, 300 | 15
228
+ | ```w2v_d2v_dbow*``` | Distributed Bag-of-Words (DBOW) | Skip-Gram (SG) | 100, 200, 300 | 15
229
+
230
+
231
+
232
+
233
+
234
+ #### Using *Word2Vec*
235
+
236
+ Installing Gensim
237
+
238
+
239
+ ```python
240
+ !pip install gensim=='3.8.3'
241
+ ```
242
+
243
+ Loading W2V (all the files for the specific model should be in the same folder)
244
+
245
+
246
+ ```python
247
+ from gensim.models import KeyedVectors
248
+
249
+ #Loading a W2V model
250
+ w2v=KeyedVectors.load('models_w2v_d2v/w2v_d2v_dm_size_100_window_15_epochs_20')
251
+ w2v=w2v.wv
252
+ ```
253
+ Viewing the first 10 entries of 'juiz' vector
254
+
255
+
256
+ ```python
257
+ w2v['juiz'][:10]
258
+ ```
259
+
260
+
261
+
262
+
263
+ array([ 6.570131 , -1.262787 , 5.156106 , -8.943866 , -5.884408 ,
264
+ -7.717058 , 1.8819941 , -8.02803 , -0.66901577, 6.7223144 ],
265
+ dtype=float32)
266
+
267
+
268
+
269
+
270
+ Viewing closest tokens to 'juiz'
271
+
272
+ ```python
273
+ w2v.most_similar('juiz')
274
+ ```
275
+
276
+
277
+
278
+
279
+ [('juíza', 0.8210258483886719),
280
+ ('juiza', 0.7306275367736816),
281
+ ('juíz', 0.691645085811615),
282
+ ('juízo', 0.6605231165885925),
283
+ ('magistrado', 0.6213295459747314),
284
+ ('mmª_juíza', 0.5510469675064087),
285
+ ('juizo', 0.5494943261146545),
286
+ ('desembargador', 0.5313084721565247),
287
+ ('mmjuiz', 0.5277603268623352),
288
+ ('fabíola_melo_feijão_juíza', 0.5043971538543701)]
289
+
290
+
291
+ #### Using *Doc2Vec*
292
+ Installing Gensim
293
+
294
+
295
+ ```python
296
+ !pip install gensim=='3.8.3'
297
+ ```
298
+
299
+ Loading D2V (all the files for the specific model should be in the same folder)
300
+
301
+
302
+ ```python
303
+ from gensim.models import Doc2Vec
304
+
305
+ #Loading a D2V model
306
+ d2v=Doc2Vec.load('models_w2v_d2v/w2v_d2v_dm_size_100_window_15_epochs_20')
307
+ ```
308
+
309
+ Inferring vector for a text
310
+
311
+
312
+ ```python
313
+ txt='direito do consumidor origem : bangu regional xxix juizado especial civel ação : [processo] - - recte : fundo de investimento em direitos creditórios'
314
+ tokens=txt.split()
315
+
316
+ txt_vec=d2v.infer_vector(tokens, epochs=20)
317
+ txt_vec[:10]
318
+ ```
319
+
320
+
321
+
322
+
323
+ array([ 0.02626514, -0.3876521 , -0.24873355, -0.0318402 , 0.3343679 ,
324
+ -0.21307918, 0.07193747, 0.02030687, 0.407305 , 0.20065512],
325
+ dtype=float32)
326
+
327
+
328
+
329
+
330
+ <a name="3.3"></a>
331
+ ### 3.3\. FastText
332
+
333
+ The FastText [4] methods, like Word2Vec, form a class of
334
+ models for creating vector representations (embeddings) for tokens. Unlike
335
+ Word2Vec, which disregards the morphology of the tokens and allocates a
336
+ different vector for each one of them, the FastText methods consider that each one
337
+ of the tokens is formed by n-grams of characters or substrings. In this way, the
338
+ representation of tokens which do not appear in the training set can be inferred
339
+ from the representation of substrings. Also, rare tokens can have more robust
340
+ representations than those returned by the Word2Vec methods.
341
+
342
+ Models are compatible with the `clean` function, but it is not necessary to use it. Remember to at least make all letters lowercase. Please check our paper or the [Gensim page](https://radimrehurek.com/gensim/models/fasttext.html) for more details. Preferably use Gensim version 4.0.1.
343
+
344
+ Below we have a summary table with some important information about the trained models:
345
+
346
+ | Filenames | FastText | Sizes | Windows
347
+ |:-------------------:|:--------------:|:--------------:|:--------------:|
348
+ | ```fasttext_cbow*``` | Continuous Bag-of-Words (CBOW) | 100, 200, 300 | 15
349
+ | ```fasttext_sg*``` | Skip-Gram (SG) | 100, 200, 300 | 15
350
+
351
+
352
+ #### Using *FastText*
353
+
354
+ installing Gensim
355
+
356
+
357
+ ```python
358
+ !pip install gensim=='4.0.1'
359
+ ```
360
+
361
+ Loading FastText (all the files for the specific model should be in the same folder)
362
+
363
+
364
+ ```python
365
+ from gensim.models import FastText
366
+
367
+ #Loading a FastText model
368
+ fast=FastText.load('models_fasttext/fasttext_sg_size_100_window_15_epochs_20')
369
+ fast=fast.wv
370
+ ```
371
+
372
+ Viewing the first 10 entries of 'juiz' vector
373
+
374
+
375
+
376
+ ```python
377
+ fast['juiz'][:10]
378
+ ```
379
+
380
+
381
+
382
+
383
+ array([ 0.46769685, 0.62529474, 0.08549586, 0.09621219, -0.09998254,
384
+ -0.07897531, 0.32838237, -0.33229044, -0.05959201, -0.5865443 ],
385
+ dtype=float32)
386
+
387
+
388
+
389
+ Viewing the first 10 vector entries of a token that was not in our vocabulary
390
+
391
+
392
+ ```python
393
+ fast['juizasjashdkjhaskda'][:10]
394
+ ```
395
+
396
+
397
+
398
+
399
+ array([ 0.02795791, 0.1361525 , 0.1340836 , -0.36824217, -0.11549155,
400
+ -0.11167661, 0.32045627, -0.33701468, -0.05198409, -0.05513595],
401
+ dtype=float32)
402
+
403
+
404
+ <a name="3.4"></a>
405
+ ### 3.4\. BERTikal
406
+
407
+
408
+ We call BERTikal our BERT-Base model (cased) [5] for Brazilian legal language. BERT models are models based on neural network architectures called Transformers. BERT models are trained with large sets of texts using the self-supervised paradigm, which is basically solving unsupervised problems using supervised techniques. A pre-trained BERT model is capable of generating representations for entire texts and can be adapted for a supervised task, e.g., text classification or question answering, using the fine-tuning mechanism.
409
+
410
+ BERTikal was trained using the Python package [Transformers](https://huggingface.co/transformers/}) in its 4.2.2 version and its checkpoint made available by us is compatible with [PyTorch](https://pytorch.org/) 1.9.0. Although we expose the versions of both packages, more current versions can be used in applications of the model, as long as there are no relevant version conflicts.
411
+
412
+ Our model was trained from the checkpoint made available in [Neuralmind’s Github repository](https://github.com/neuralmind-ai/portuguese-bert) by the authors of recent research [6].
413
+
414
+ #### Using *BERTikal*
415
+
416
+ Installing Torch e Transformers
417
+
418
+
419
+ ```python
420
+ !pip install torch=='1.8.1' transformers=='4.2.2'
421
+ ```
422
+
423
+ Loading BERT (all the files for the specific model should be in the same folder)
424
+
425
+
426
+ ```python
427
+ from transformers import BertModel, BertTokenizer
428
+
429
+ bert_tokenizer = BertTokenizer.from_pretrained('model_bertikal/', do_lower_case=False)
430
+ bert_model = BertModel.from_pretrained('model_bertikal/')
431
+ ```
432
+
433
+ --------------
434
+
435
+ <a name="4"></a>
436
+ ## 4\. Demonstrations
437
+
438
+ For a better understanding of the application of these models, below are the links to notebooks where we apply them to a legal dataset using various classification models such as Logistic Regression and CatBoost:
439
+
440
+ - **BERT notebook** :
441
+ [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/felipemaiapolo/legalnlp/blob/main/demo/BERT/BERT_TUTORIAL.ipynb)
442
+
443
+ - **Word2Vec notebook** :
444
+ [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/felipemaiapolo/legalnlp/blob/main/demo/Word2Vec/Word2Vec_TUTORIAL.ipynb)
445
+
446
+ - **Doc2Vec notebook** :
447
+ [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/felipemaiapolo/legalnlp/blob/main/demo/Doc2Vec/Doc2Vec_TUTORIAL.ipynb)
448
+
449
+
450
+
451
+ --------------
452
+
453
+ <a name="5"></a>
454
+ ## 5\. References
455
+
456
+ [1] Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean, J. (2013b).
457
+ Distributed representations of words and phrases and their compositionality.
458
+ In Advances in neural information processing systems, pages 3111–3119.
459
+
460
+ [2] Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013a). Efficient estimation of
461
+ word representations in vector space. arXiv preprint arXiv:1301.3781.
462
+
463
+ [3] Le, Q. and Mikolov, T. (2014). Distributed representations of sentences and
464
+ documents. In International conference on machine learning, pages 1188–1196.
465
+ PMLR.
466
+
467
+ [4] Bojanowski, P., Grave, E., Joulin, A., and Mikolov, T. (2017). Enriching
468
+ word vectors with subword information. Transactions of the Association for
469
+ Computational Linguistics, 5:135–146.
470
+
471
+ [5] Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training
472
+ of deep bidirectional transformers for language understanding. arXiv preprint
473
+ arXiv:1810.04805.
474
+
475
+ [6] Souza, F., Nogueira, R., and Lotufo, R. (2020). BERTimbau: pretrained BERT
476
+ models for Brazilian Portuguese. In 9th Brazilian Conference on Intelligent
477
+ Systems, BRACIS, Rio Grande do Sul, Brazil, October 20-23
478
+