KAMATERI E., STAMATIS V., DIAMANTARAS K. & SALAMPASIS M. (2022). Automated single-label patent classification using ensemble classifiers. In *2022 14th International Conference on Machine Learning and Computing (ICMLC)*, p. 324–330.

LEE J.-S. & HSIANG J. (2019). Patentbert: Patent classification with fine-tuning a pre-trained bert model. *arXiv preprint arXiv:1906.02124*.

LI S., HU J., CUI Y. & HU J. (2018). Deeppatent: Patent classification with convolutional neural networks and word embedding. *Scientometrics*, **117**(2), 721–744. DOI: 10.1007/s11192-018-2905-5.

LIU Y., OTT M., GOYAL N., DU J., JOSHI M., CHEN D., LEVY O., LEWIS M., ZETTLEMOYER L. & STOYANOV V. (2019). Roberta: A robustly optimized bert pretraining approach. *arXiv preprint arXiv:1907.11692*.

MARTIN L., MULLER B., ORTIZ SUÁREZ P. J., DUPONT Y., ROMARY L., DE LA CLERGERIE É. V., SEDDAH D. & SAGOT B. (2020). Camembert: a tasty french language model. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*.

MIHALCEA R. & TARAU P. (2004). Textrank: Bringing order into text. In *Proceedings of the 2004 conference on empirical methods in natural language processing*, p. 404–411.

NIU M. & CAI J. (2019). A label informative wide & deep classifier for patents and papers. In *Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)*, p. 3438–3443.

PIRES T., SCHLINGER E. & GARRETTE D. (2019). How multilingual is multilingual bert? *arXiv preprint arXiv:1906.01502*.

PIROI F. (2010). Clef-ip 2010: Retrieval experiments in the intellectual property domain.

PIROI F. (2021). The marec/irec data set. DOI: 10.48436/2zx6e-5pr64.

PIROI F., LUPU M., HANBURY A. & ZENZ V. (2011). Clef-ip 2011: Retrieval in the intellectual property domain.

QIU X., HUANG X.-J., LIU Z. & ZHOU J. (2011). Hierarchical text classification with latent concepts. In *Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies*, p. 598–602.

RADAUER A. & WALTER L. (2010). Elements of good practice for providers of publicly funded patent information services for smes–selected and amended results of a benchmarking exercise. *World Patent Information*, **32**(3), 237–245.

RAFFEL C., SHAZEER N., ROBERTS A., LEE K., NARANG S., MATENA M., ZHOU Y., LI W. & LIU P. J. (2020). Exploring the limits of transfer learning with a unified text-to-text transformer. *The Journal of Machine Learning Research*, **21**(1), 5485–5551.

RISCH J., GARDA S. & KRESTEL R. (2020). Hierarchical document classification as a sequence generation task. In *Proceedings of the Joint Conference on Digital Libraries (JCDL)*, p. 147–155.

RISCH J. & KRESTEL R. (2019). Domain-specific word embeddings for patent classification. *Data Technologies and Applications*.

RODA G., TAIT J., PIROI F. & ZENZ V. (2009). Clef-ip 2009: Retrieval experiments in the intellectual property domain. volume 1175, p. 385–409. DOI: 10.1007/978-3-642-15754-7_47.

ROUDSARI A. H., AFSHAR J., LEE W. & LEE S. (2021). Patentnet: multi-label classification of patent documents using deep learning based language understanding. *Scientometrics*.
SUÁREZ P. J. O., SAGOT B. & ROMARY L. (2019). Asynchronous pipeline for processing huge corpora on medium to low resource infrastructures. In *7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7)* : Leibniz-Institut für Deutsche Sprache.

VERBERNE S. & D'HONDT E. (2011). Patent classification experiments with the linguistic classification system lcs in clef-ip 2011. volume 1177.

WU C.-H., KEN Y. & HUANG T. (2010). Patent classification system using a new hybrid genetic algorithm support vector machine. *Applied Soft Computing*, **10**(4), 1164–1177. Optimisation Methods Applications in Decision-Making Processes, DOI: https://doi.org/10.1016/j.asoc.2009.11.033.

YANG Z., DAI Z., YANG Y., CARBONELL J., SALAKHUTDINOV R. & LE Q. V. (2020). Xlnet : Generalized autoregressive pretraining for language understanding.

YUN J. & GEUM Y. (2020). Automated classification of patents : A topic modeling approach. *Computers & Industrial Engineering*, **147**, 106636.

ZHANG L., LIU W., CHEN Y. & YUE X. (2022). Reliable multi-view deep patent classification. *Mathematics*, **10**(23), 4545.

ZUO Y., MOUZOUN H., DOUDANE S. G., GERDES K. & SAGOT B. (2022). Patent classification using extreme multi-label learning : A case study of french patents. In *SIGIR 2022-PatentSemTech workshop-3rd Workshop on Patent Text Mining and Semantic Technologies*.
# A Important Features from Sparse Classifiers

We utilized a Logistic Regression model with TF-IDF sparse features, in which each category corresponds to a classifier. In our data preprocessing step, we first removed stop words and then lemmatized each remaining word. We set the dimensionality of the features to 10,000 and excluded words that occurred in more than 90% of the documents. The most important features are represented by the coefficients in the z-equation of the logistic regression, denoted by $w_1$ to $w_n$ in the following equation :

$$y = \frac{1}{1 + e^{-z}}$$
$$z = w_0 + w_1x_1 + w_2x_2 + w_3x_3 + \dots + w_nx_n$$

To select the most important features for each classifier, we took the top five features with the highest coefficient values. We then identified the 100 words that appeared most frequently across all classifiers and added them to our final encoder. These words will be used to improve the model's accuracy in classification tasks.

<table><caption>TABLE 9 – Examples of important features of logistic regression models trained on different patent parts</caption><thead><tr><th>title+abstract</th><th>claims</th><th>description</th></tr></thead><tbody><tr><td>composition</td><td>outil</td><td>moteur</td></tr><tr><td>moteur</td><td>pourcent</td><td>véhicule</td></tr><tr><td>outil</td><td>composition</td><td>outil</td></tr><tr><td>composé</td><td>moteur</td><td>eau</td></tr><tr><td>machine</td><td>véhicule</td><td>composé</td></tr><tr><td>roue</td><td>formule</td><td>polymère</td></tr><tr><td>eau</td><td>machine</td><td>roue</td></tr><tr><td>signal</td><td>atome</td><td>composition</td></tr><tr><td>formule</td><td>dispositif</td><td>fibre</td></tr><tr><td>polymère</td><td>signal</td><td>signal</td></tr><tr><td>véhicule</td><td>roue</td><td>combustion</td></tr><tr><td>combustion</td><td>groupe</td><td>machine</td></tr><tr><td>fabrication</td><td>couche</td><td>gaz</td></tr><tr><td>produit</td><td>acide</td><td>acide</td></tr><tr><td>gaz</td><td>polymère</td><td>air</td></tr><tr><td>air</td><td>gaz</td><td>électrique</td></tr><tr><td>commande</td><td>air</td><td>atome</td></tr><tr><td>fibre</td><td>eau</td><td>mesure</td></tr><tr><td>fluide</td><td>combustion</td><td>fluide</td></tr><tr><td>mesure</td><td>fibre</td><td>piston</td></tr></tbody></table>
## B Visualization of IPC distributions

Given that different countries have varying priorities in protecting different technical fields, it is important to consider the distance between training and testing data when performing patent classification. Simply adding more data for training without considering the distribution of the data across countries may lead to suboptimal classification results. To demonstrate this phenomenon, we visualize the distribution of labels at the IPC subclass level in Figure B.

<figure><img src="image_84.png" /><figcaption>FIGURE 3 – Distribution of labels per IPC section.</figcaption></figure>
## C Machine Translation en-fr

In order to improve the classification of French patents, we used a T5 (Raffel et al., 2020) model that was trained on aligned English-French data. Specifically, we leveraged the first release of the EuroPat dataset (Heafield et al., 2022), restricting our analysis to patents that were published after 2010. This allowed us to compile a corpus of 2 million sentence pairs for training our model.

We drew inspiration from (Jehl & Riezler, 2018) to investigate the effectiveness of incorporating special tokens to introducing information as patent sections (\<A\>, \<B\>,...,\<H\>) or text types (\<title\>, \<abstract\>, \<claims\>, \<description\>). Our study compares various methods to determine the optimal approach.

For each approach, we fine-tuned the t5-base with 220 million parameters for a single epoch, using a batch size of 16 and a learning rate of 1e-4. The maximum input and output lengths were set to 256. This configuration was selected to optimize the balance between training time and model performance. To evaluate the performance of our model, we relied on the widely used machine translation metric, BLEU score.

<table><caption>TABLE 10 – Performances of translation models.</caption><thead><tr><th></th><th>BLEU</th></tr></thead><tbody><tr><td>Original Text</td><td>79.15</td></tr><tr><td>IPC1</td><td>79.10</td></tr><tr><td>Text Type</td><td>78.85</td></tr><tr><td><strong>IPC1, Text Type</strong></td><td><strong>79.19</strong></td></tr></tbody></table>

We can see that the translation model achieves the best performance when it differentiates between text types and text domains. Therefore, in our main experimental results, we demonstrate the use of the model with special tokens of IPC1s and text types during training for translation and data augmentation.
<figure><img src="image_85.png" /></figure>