[image] [image] Ressource protégée - Archive ouverte HAL Accéder directement au contenu

*[image] [image]*

• Documentation

• FR
Français (FR)

• Anglais (EN)
Se connecter

• HAL science ouverte

• Recherche
Loading...

## Recherche avancée

Information de documents

Titres

○ Titres

○ Sous-titre

○ Titre de l'ouvrage

○ Titre du volume (Série)

○ Champ de recherche par défaut (multicritères) + texte intégral des PDF

○ Résumé

○ Texte intégral indexé des documents PDF

○ Mots-clés

○ Type de document

○ Sous-type de document

○ Tous les identifiants du document

○ Identifiant HAL du dépôt

○ Langue du document (texte)

○ Pays (Texte)

○ Ville

○ À paraître (true ou false)

Ajouter

Auteur
<figure><img src="image_1.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
## Auteur (multicritères)

○ Auteur (multicritères)

○ Auteur : Nom complet

○ Auteur : Nom de famille

○ Auteur : Prénom

○ Auteur : Complément de nom, deuxième prénom

○ Auteur : Organisme payeur

○ Auteur : IdHal (chaîne de caractères)

○ Auteur : Fonction

○ Auteur : personID (entier)

○ Auteur : Identifiant de l'organisme payeur

○ Auteur : Identifiant de la structure

○ Directeur de thèse

○ Éditeur

○ Éditeur scientifique

○ Éditeur de la série

## Ajouter

### Structure

### Structure (multicritères)

○ Structure (multicritères)

○ Structure : Acronyme

○ Structure : Nom

○ Structure : Code

○ Structure : Pays

○ Structure : Type

○ Structure : État dans le référentiel

○ Structure : Identifiant HAL de la structure

○ Structure/équipe de recherche : Pays

○ Structure/regroupement d'équipes : Pays

○ Structure/laboratoire : Pays

○ Structure/regroupement de laboratoires : Pays

○ Structure/institution : Pays

○ Structure/regroupement d'institutions : Pays

## Ajouter

## Autres
# Exploitation of causal relation for automatic extraction of contradiction from a domain-restricted patent corpus

Daria Berdyugina¹ and Denis Cavallucci¹

¹ICUBE/CSIP, INSA of Strasbourg, 24 Boulevard de la Victoire, 67084, Strasbourg, France
dberdyugina@etu.unistra.fr, denis.cavallucci@insa-strasbourg.fr

**Abstract.** Altshuller contradiction matrix is one of the most popular tools among TRIZ practitioners, especially beginners, due to its simplicity and intuitive design. However, scientific and technological progress induces the constant appearance of new scientific vocabulary, which lower accuracy when using this static tool from the end of the sixties. Some attempts to rebuild the matrix or update it has been made within the past four decades but without any successful legitimation due to the lack of scientific proof regarding its relevance. Our recent findings in the use of Natural Language Processing (NLP) techniques allow the creation of a methodology for automatic extraction of the necessary information for establishing a domain-restricted contradiction matrix. In this paper, we relate a technique that exploits the internal language semantic structure to mine the causal relation between terms in patent texts. Moreover, the subject or domain restriction for a patent collection allows observing the links between extracted information at the over-text level. Such an approach relies on inter-and extra-textual features and permits a real-time extraction of contradictory relations between elements. These extracted elements could be presented in matrix form, inspired by The Altshuller contradiction matrix. We postulate that such a representation allows the construction of a state of the art in each domain, which will facilitate the use of TRIZ to solve contradictions within it.

**Keywords:** TRIZ, NLP, contradiction matrix, text-mining, automatic extraction

## 1 Introduction

The Contradiction Matrix (CM) remains a popular tool among a lot of TRIZ practitioners because of its simplicity. Despite the existence of other tools developed by the experts (for example, ARIZ85C [1] or Vepole [2]), the CM does not lose popularity. Created in 1969, this tool is largely used, notably, by less highly experienced TRIZ users.

Nonetheless, due to today's rapid scientific advancements, the tool's efficiency is called into question. Since the establishment of Altshuller's matrix, many new technologies, methods, and even whole fields of study have evolved. The CM is out of date because the terminology is becoming outdated. Another issue is its general nature. For newcomers to TRIZ, forming a relevant contradiction and connecting termi-
Champ de recherche par défaut (multicritères)

○ <u>Champ de recherche par défaut (multicritères)</u>

○

○ <u>Revues (multicritères)</u>

○ <u>Revue : Éditeur</u>

○ <u>Revue : Titre abrégé</u>

○ <u>Revue : Titre</u>

○ <u>Revue : Identifiant interne</u>

○ <u>Revue : Date de début de publication</u>

○ <u>Revue : ISSN électronique</u>

○ <u>Revue : ISSN</u>

○ <u>Revue : Éditeur</u>

○ <u>Revue : Couleur dans SHERPA/RoMEO</u>

○ <u>Revue : État dans le référentiel</u>

○

○ <u>Colloque (multicritères)</u>

○ <u>Colloque : Titre</u>

○ <u>Colloque : Organisateur</u>

○ <u>Colloque : date de début (Année)</u>

○ <u>Colloque : date de fin (Année)</u>

○ <u>Conférence invitée (oui/non)</u>

○

○ <u>Projets ANR (multicritères)</u>

○ <u>Projet ANR : Acronyme</u>

○ <u>Projet ANR : Acronyme du programme</u>

○ <u>Projet ANR : Code décision (référence)</u>

○ <u>Projet ANR : Nom</u>

○ <u>Projet ANR : Identifiant interne</u>

○ <u>Projet ANR : État dans le référentiel</u>

○

○ <u>Projets européens (multicritères)</u>

○ <u>Projet européen : Acronyme</u>

○ <u>Projet européen : Identifiant de l'appel à projet</u>

○ <u>Projet européen : Référence</u>

○ <u>Projet européen : Nom</u>

○ <u>Projet européen : Date de fin</u>

○ <u>Projet européen : Financement</u>

○ <u>Projet européen : Date de début</u>

○ <u>Projet européen : État dans le référentiel</u>
nology from their field of expertise to technical data utilized in the CM might be chal-
lenging, resulting in a misinterpretation of its use.

The IDM (Inventive Design Method) was developed to overcome some of TRIZ's
drawbacks, one of which was the lack of a codified ontology. Another disadvantage
of the IDM-resolved grounding theory is the difficulty of performing any computation
on abstract ideas [3]. Problems, partial solutions, and parameters are the three funda-
mental core principles of IDM. However, focusing on the parameters is required for
CM reconstruction on the user's patent corpus because they provide summarized in-
formation about the technologies specified in the patents. Furthermore, the parameters
define CM's "borders."

As previously stated, the CM's author examined around 40,000 of the most innova-
tive patent texts [4]. The ability of automatic text-mining tools has become crucial in
recent years. Patents are a plentiful and numerically accessible source of inventive
data. As a result, we intend to make use of a corpus of patent texts from the most re-
cent field of research to develop a CM based on the user's selection of patents. Patent
analyzing apps are becoming increasingly popular among industry and engineers, thus
it's critical to identify appropriate methodologies and processing tools. In any patent-
related operation, the better the processing instrument employed, the better the results.

Because of their extensive and complex language structures and unusual style,
patent documents are frequently difficult to comprehend. This is owing to the patent
text's dual nature, which is both a legal and a technical document aimed at protecting
the inventor and identifying the invention's limitations. As a result, given the large
number of patents issued each day, it is more practical for businesses and scientists to
have a more compact TRIZ-based representation form for patents.

The standards for doing an automatic patent analysis are exceedingly high. It ne-
cessitates some knowledge of domain-specific technologies and information retrieval
methods. This type of analysis is both expensive and difficult to perform. It's difficult
to do such a common analysis scenario by hand.

However, the modern approaches and techniques of Computer Science and NLP
permits performing such analysis and extraction faster. Notably, the transformers, a
Deep Learning model that employs the self-attention process, differentially ranks the
significance of each component of the input data (for example, BERT<fnref n="1" /> [5]), facilitates
the task of adaptation of the model for a specific task.

Our research goal is to determine the best method for automatically extracting pa-
rameters from domain-specific corpus to populate the CM's rows and columns.

By analyzing the text of patents, we concluded that the contradictions, expressed in
texts of patents are usually not fully disclosed, i.e., only one parameter of a contradic-
tion is clearly expressed, but the second parameter is either hidden or located else-
where in the text. However, this second parameter could be found thanks to the inter-
nal logic of the text structure.

In this article, we present an overview of IDM approaches, as well as a literature
analysis on transformers' learning methodologies (2). Following that, we describe a
method that is developed for the automatic extraction of IDM-related information

<footnote n="1">¹ Bidirectional Encoder Representations from Transformers (BERT)</footnote>
○ Projet européen : Identifiant interne

O

○ Date de production : année

○ Date de mise en ligne : année

○ Date de publication : année

○ Date d'écriture : année

○ Date de modification du dépôt : année

○ Date de dépôt : année

○ Date de publication électronique : année

O

○ Collection HAL (multicritères)

Collection HAL : catégorie

○ Collection HAL : Code

○ Collection HAL : Nom

○ Collection HAL : Identifiant interne

O

○ Identifiant interne du contributeur/déposant

○ Nom complet du contributeur/déposant

O

○ Domaines

○ Domaine primaire

○ Domaine racine

○ Sous-domaine niveau 1

○ Sous-domaine niveau 2

○ Sous-domaine niveau 3

O

○ Statut du document

○ Version du document

○ Type de dépôt

○ Type de document

○ ISBN

○ Numéro - référence

○ Identifiant DOI

O

○ Classification

○ Audience

○ Vulgarisation

○ Comité de lecture - texte (oui ou non)

○ Actes de colloque

○ Référence interne
from patents, particularly the parameters (3). Then, employing the approaches, men-
tioned above, we outline the methodology for improving and refining the process of
extracting parameters (4). The findings of our experimental efforts are then presented
in (5).

2 State of Art

This chapter discusses the TRIZ, Inventive Design Method ideas, and NLP methods
used to attain our goal to better understand the methodology of the current research.

2.1 TRIZ

TRIZ [6], [7] arose from a careful examination of a hundred thousand patent docu-
ments. According to Genrich Altshuller, the birth of innovation is conditional on the
presence of objective rules and the observance of universal principles.

His investigation led him to the conclusion that some patterns of difficulties repeat
irrespective of industrial domains, regardless of the industry to which a specific arti-
fact belongs.

Genrich Altshuller also distinguishes between a tiny amplitude invention, which
may be characterized as a modest innovation requiring little work, and discoveries
that need significantly more time and effort to obtain. His descriptions of the trial-
and-error approach are tied to these efforts.

This progression from low-amplitude innovation to scientific discovery has re-
sulted in the establishment of five levels of inventiveness [8].

TRIZ incorporates a variety of essential principles that may be used to conceive
and solve any problems, as well as obtain a solution without compromising.

The most important of these principles are the rules of Engineering Systems Evo-
lution, contradiction, resources, and ideality [9].

In the context of TRIZ, contradiction is the basis of any problem formulation.

This term reflects the idea that increasing the value of one parameter leads to a de-
crease in the value of another parameter in a system [10]. It is important to emphasize
that, according to TRIZ, any inventive problem must be expressed as a contradiction.

Further to that, the problem is described generally, and the search for an uncom-
promising solution to this contradiction is done through meditation by analogy utiliz-
ing the innovative concepts of TRIZ [11].

## 2.2 Inventive Design Method

The current study aims to extract IDM-related information from a domain-specific
corpus. Parameters are of particular significance to us because they reflect the ele-
ments of contradiction. We have discussed the important topics above for explanation
purposes:
issues, partial solutions, parameters, and contradictions.

A major part of our laboratory's work is based on TRIZ, which was invented by
Genrich Altshuller.
○ Financement

○ Collaborations

Ajouter

Lancer la recherche

Recherche experte (SolR)

Recherche experte (SolR)

Lancer la recherche

Vers la recherche avancée

• Déposer

X

X

X

Loading...

X

Le document que vous demandez est en access restreint
jusqu'au 2025-01-01 !

Revenir a la notice

Retourner à l'accueil

[image]

[image]

[image]

[image]

Contact

• Support

Ressources

• Documentation

• FAQ

• API
The TRIZ-based Inventive Design Method (IDM) expands on the limitations of the grounding theory, notably the absence of a defined ontology, which makes it impossi- ble to perform any computation on its abstract concepts. Thanks to IDM, this ontol- ogy has been created.

The problem-solving process, according to IDM, consists of four steps [12]:

1. Information extraction, comprising “problems” and “partial solutions”;

2. Contradiction formulation;

3. Solving each key contradiction;

4. Selection of the most suitable solution concept.

For making TRIZ effective for industrial innovation, the IDM provides a functional formulation of the contradiction notion. The contradiction, according to this defini- tion, is “[...] characterized by a set of three parameters and where one of the parameters can take two possible opposite values $Va$ and $\overline{Va}$” [10]. Hence, it is nec- essary to provide a definition for two types of parameters.

There are two types of parameters: action parameters (AP) and evaluation parame- ters (EP).

The first, AP, “[...] is characterized by the fact that it has a positive effect on an- other parameter when its value tends towards and that it harms another parameter when its value tends towards [...]. (i.e., in the opposite direction)” [10].

The EP “[...] can evolve under the influence of one or more action parameters” and which allows to “evaluate the positive aspect of a choice made by the designer” [10].

The graphical representation of the contradiction notion is provided below.

$$AP \frac{Va}{\overline{Va}} \begin{pmatrix} EP_1 & EP_2 \\ -1 & 1 \\ 1 & -1 \end{pmatrix} \quad (1)$$

During the application of the solution to the technical contradiction, the physical contradiction may arise.

Physical contradiction reveals a system characteristic when the application of ac- tion in a physical condition results in contradictory requirements for its opposite char- acteristic [13].

## 2.3 Transformer model

Transformers [14] have taken NLP by storm since its introduction in 2017, delivering greater parallelization and better modeling of long-term dependencies.

Transformers, like recurrent neural networks (RNNs), are built to handle sequential input data, such as natural language, for tasks like translation. This type of model does not always treat the input data in order. On the other hand, the attention mechanism permits obtaining the context for any place of the input phrase. For example, for the text input, it is not needed for such a model to process the beginning of the phrase be-
• OAI-PMH

• AuréHAL

## Informations

• À propos

• Données personnelles

• Mentions légales

• Accessibilité

• Conformité RGAA

## Questions juridiques

• Je publie, quels sont mes droits ?

• Loi pour une République numérique

• Stratégie de non-cession des droits

• SHERPA/RoMEO

## Portails

• Portails institutionnels

• HAL

• HAL SHS

• HAL Thèses

• MédiHAL

## CCSD

• CCSD

• Episciences

• Sciencesconf
fore the end of its phrase. I.e., it determines the context that provides the meaning for each word in a phrase. This feature allows shortening the training duration [14].

Introduced in 2017, this model starts to replace recurrent neural networks models (for example, long short-term memory), notably for NLP tasks [15]. The main feature of transformers, additional parallelisation, permits training on larger datasets than previous models. Hence, the pre-trained systems (for example, BERT or GPT<fnref n="2" /> that are trained on large datasets [16]) have been created. The architecture of such systems allows a fine-tuning for a specific NLP task.

BERT [5] is the most well-known Transformer-based model; it achieved cutting-edge performance in a variety of benchmarks and remains a must-have baseline [17].

The standard BERT workflow consists of two stages: pre-training and fine-tuning. Pre-training tasks include masked language modeling (MLM, which predicts randomly masked input tokens) and next sentence prediction (NSP, which predicts if two input phrases are close to each other). One or more fully-connected layers are generally placed on top of the final encoder layer to fine-tune for downstream applications [17]. It is important to note that there are a lot of versions of BERT, provided by Google<fnref n="3" /> and HuggingFace [15]. These versions include ‘large’ and ‘base’ variants, which are differentiated by numbers, layers, and hidden state size.

# 3 Methodology

In this chapter, we describe the methodology applied to achieve the goal of automatic extraction of inventive information (parameters and their probability to be in contradiction) from the domain-specific patent corpus.

## 3.1 General description of Methodology

By analyzing the text of patent documents, we concluded that two parameters in contradiction are linked by the causal relation. For example:

> The mold has an excellent thermal conductivity since it is formed from a metal material. When a mold of a metal material is used and when the mold temperature is set at a level considerably lower than the deflection temperature under load of a crystalline thermoplastic resin as described above, the molten crystalline thermoplastic resin filled in the cavity begins to be cooled as soon as it is brought into contact with the cavity wall of the mold. As a result, an amorphous layer or a low-crystallinity fine crystal layer is formed as a surface of the molded article. Such a layer is generally called a skin layer. The molded article having the skin layer has a problem in that it is greatly degraded in surface properties.[18]

In this example, we could find an expressed contradiction between excellent thermal conductivity and degraded surface properties, which are two EP. The AP here is the mold. These two EPs are located far from each other. However, there is an inter-textual link between them, which is asserted by causal relations. I.e., when the AP is

<footnote n="2">² Generative Pre-trained Transformer</footnote>

<footnote n="3">³ https://github.com/google-research/bert</footnote>
expressed as a cause that influences two EPs (effects) and when one of the EP has a negative connotation and the second EP has a positive connotation, in this situation we could suppose that two effects are EPs and they are in a contradiction situation. Fig. 1 illustrated the described idea.

<figure><img src="image_2.png" /><figcaption>Fig. 1. Graphical illustration of the approach.</figcaption></figure>

For the task of polarity calculation, we employ the Flair model [19], which was trained to detect the polarity of the phrases.

## 3.2 Extraction model

In order to perform the above-mentioned extraction, it is needed to fine-tune a model, which could be capable to recognize the causal relation. As it is described in the previous section, we employ the transformer model (BERT and ALBERT) in order to extract the domain-specific parameters and their probability to be in contradiction. In order to perform this task, it is needed to apply a fine-tuning step.

For performing this step, we took an annotated corpus for the classification of semantic relation between pairs of nominals [20]. This corpus<fnref n="4" /> contains not only annotation for causal relations, but also a lot of semantic relations, such as Component-Whole or Member-Collection. The example of annotation is presented in Table 1.

<table><caption>Table 1. Example of the original annotated corpus.</caption><thead><tr><td>sentence (string)</td><td>relation (class label)</td></tr></thead><tbody><tr><td>The &lt;e1&gt;burst&lt;/e1&gt; has been caused by water hammer &lt;e2&gt;pressure&lt;/e2&gt;.</td><td>1 (Cause-Effect(e2,e1))</td></tr></tbody></table>

However, for our research, it is important to extract not only the nominals but also the adjective, which could define the polarity of a parameter. Thus, we were obliged to re- annotate this corpus, considering only the Cause-Effect part.

<table><tr><td>Sentence: 1,The,DT,N</td></tr><tr><td>Sentence: 1,burst,NN,E</td></tr><tr><td>Sentence: 1,has,VBZ,N</td></tr><tr><td>Sentence: 1,been,VBN,N</td></tr></table>

<footnote n="4">⁴ Accessible in https://huggingface.co/datasets/sem_eval_2010_task_8.</footnote>
<table><caption>**Table 1.** Example of the re-annotated corpus.</caption><tr><td>Sentence: 1,caused,VBN,N</td></tr><tr><td>Sentence: 1,by,IN,N</td></tr><tr><td>Sentence: 1,water,NN,C</td></tr><tr><td>Sentence: 1,hammer,NN,C</td></tr><tr><td>Sentence: 1,pressure,NN,C</td></tr></table>

## Classification of the contradictions

By performing the first experiments with the extraction model, we infer that the quantity of extracted parameters exceeds the limitation of the understanding, i.e., there is a lot of noise, expressed by the extracted terms, which are not the parameters.

In order to overcome this difficulty, we apply another model for contradictions classification.

Using the output of the causal extraction, we annotate 7,268 pairs of candidates. In Table 1, we present an example of the annotated corpus, where label 0 represents the absence of contradiction and label 1 represents the presence of contradiction.

<table><caption>**Table 1.** Example of annotation for sentence-pair classification task</caption><thead><tr><th>idx</th><th>label</th><th>sentence1</th><th>sentence2</th></tr></thead><tbody><tr><td>Num 18</td><td>0</td><td>'occurrence of sink mark'</td><td>'knock'</td></tr><tr><td>Num 19</td><td>1</td><td>'occurrence of sink mark'</td><td>'excellent mass productivity'</td></tr></tbody></table>

# 4 Case Study

In this section, we present the application of our approach in a concrete case study.

To validate our methodology, we have been using a set of 10 patents, having the subject of the creation of hydrophobic paper. The corpus statistic is presented in Table 1.

<table><caption>**Table 1.** Corpus statistics</caption><thead><tr><th>Number of patents</th><th>Words</th><th>Symbols</th></tr></thead><tbody><tr><td>10</td><td>127,612</td><td>775,121</td></tr></tbody></table>

Thanks to our extraction model, we extract 1,051 pairs of contradictions candidates. However, by applying the second model of contradictions classifications, we achieve to restrict this number by 28 candidates. We restricted the score of the probability by > 0.45. The score is relatively low because the classification model is trained based on another domain corpus. Based on a restricted set of candidates, the experts approve 17 pairs of contradictions.

However, in the list of extracted pairs not classified as a potential contradiction, there is a set of pairs that have been approved by the experts to be in contradiction.
Our methodology has been built in an API that allows users to examine a domain-
specific corpus of patent documents to construct the matrix representation of opposing
parameters. The work is still in progress.

5 Discussion and Results Evaluation

In the present section, we discuss the results of the implementation of our methodology.

In the context of classification of extracted contradictions, our approach has an im-
portant drawback which is manifested in the presence of the noise. However, with
more cases of the use of our methodology, we aim to collect more examples of valid
contradiction pairs.

The extraction of pairs of candidates has equally its drawbacks. For the present
day, we did not test other models for causal relation mining, for example, RoBERTa
[21]. Moreover, we still lack annotated data from patent texts.

However, the present results remain promising in terms of parameters and contra-
dictions identification. As our experiment shows, we achieve correctly identify 17
contradictions from 28.

It is important to note that the parameters, identified thanks to our method, could
not be in contradiction with others parameters. That is the reason, why we decide to
replace the classical matrix representation for our tool with a chord diagram. The ex-
ample of reviewed CM is shown in Fig. 1. The base for this diagram is the list of vali-
dated pairs of parameters from the case study.
<figure><img src="image_3.png" /><figcaption>Fig. 1. Example of reviewed representation of CM</figcaption></figure>

## 6 Conclusion

In the present article, we introduce the method for the automatic extraction of contradictions from the domain-specific patent corpus. Our methodology is based on IDM, for which a contradiction is one of the core elements in the inventive problem-solving process.

On a higher scale, our approach attempts to simplify the problem-solving process by automating the extraction of each required element in order to populate the ontology. The automated method not only allows each TRIZ-user to save time, and to gain a greater representative content from all corpus data.

Presently, we are working on dressing the “borders” of the CM. However, for future work, we consider linking the extracted contradictions to any source permitting solving them, notably the inventive principles. Moreover, the source of solution could be found in the claims of patent texts or any other sources of inventive information, such as scientific articles.

In a longer perspective, we intend to use additional sources of innovative information, such as scientific articles, to fill in the gaps left by other sources of extraction.
References

[1] J. Marconi, "ARIZ : The Algorithm for Inventive Problem Solving," *Triz J.*, Apr. 1998, Accessed: Apr. 08, 2021. [Online]. Available: https://triz-journal.com/ariz-algorithm-inventive-problem-solving/

[2] S. Dubois, P. Lutz, F. Rousselot, and E. Caillaud, "A Formal Model for the Representation of Problems Based on TRIZ," in *International Conference on Engineering Design, ICED 05*, Melbourne, Australia, Aug. 2005, p. NA. [Online]. Available: https://hal.archives-ouvertes.fr/hal-00340989

[3] A. Souili and D. Cavallucci, "Automated Extraction of Knowledge Useful to Populate Inventive Design Ontology from Patents," in *TRIZ – The Theory of Inventive Problem Solving*, Springer, 2017, pp. 43–62. doi: 10.1007/978-3-319-56593-4_2.

[4] G. Altshuller, *40 Principles: TRIZ Keys to Innovation*. Technical Innovation Center, Inc., 2002.

[5] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding," *ArXiv*181004805 Cs, May 2019, Accessed: Apr. 29, 2022. [Online]. Available: http://arxiv.org/abs/1810.04805

[6] G. Altshuller, *Creativity As an Exact Science*. Taylor & Francis, 1984.

[7] G. Altshuller and G. Altov, *And suddenly the inventor appeared: TRIZ, the theory of inventive problem solving*. Technical Innovation Center, Inc., 1996.

[8] V. Souchkov, "Differentiating Among the Five Levels of Solutions," *Online TRIZ J.*, 2007.

[9] D. Cavallucci, "Contribution à la conception de nouveaux systemes mécaniques par integration methodologique," These de doctorat, Université Louis Pasteur (Strasbourg) (1971-2008), 1999. Accessed: Apr. 07, 2021. [Online]. Available: http://www.theses.fr/1999STR13238

[10] F. Rousselot, C. Zanni-Merk, and D. Cavallucci, "Towards a formal definition of contradiction in inventive design," *Comput. Ind.*, vol. 63, no. 3, pp. 231–242, 2012, doi: https://doi.org/10.1016/j.compind.2012.01.001.

[11] B. Zlotin, A. Zusman, L. Kaplan, S. Visnepolschi, V. Proseanic, and S. Malkin, "TRIZ beyond technology: The theory and practice of applying TRIZ to nontech- nical areas," *TRIZ J.*, vol. 6, no. 1, 2001.

[12] D. Cavallucci, "From TRIZ to Inventive Design Method (IDM): towards a for- malization of Inventive Practices in R&D Departments," in *Japan TRIZ Symposium 2012*, 2012, p. 2.

[13] D. Mann, "Re-Thinking Physical Contradictions #1: Technical Problems," *Triz J.*, Mar. 2018, Accessed: Apr. 08, 2021. [Online]. Available: https://triz-journal.- com/re-thinking-physical-contradictions-1-technical-problems/

[14] A. Vaswani *et al.*, "Attention is All you Need," in *Advances in Neural Information Processing Systems*, 2017, vol. 30. [Online]. Available: https://proceed- ings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf

[15] T. Wolf *et al.*, "Transformers: State-of-the-Art Natural Language Processing," in *Proceedings of the 2020 Conference on Empirical Methods in Natural Language*
*Processing: System Demonstrations*, Online, Oct. 2020, pp. 38–45. doi:
10.18653/v1/2020.emnlp-demos.6.

[16] “Open Sourcing BERT: State-of-the-Art Pre-training for Natural Language Pro-
cessing,” *Google AI Blog*. http://ai.googleblog.com/2018/11/open-sourcing-bert-
state-of-art-pre.html (accessed Apr. 30, 2022).

[17] A. Rogers, O. Kovaleva, and A. Rumshisky, “A Primer in BERTology: What we know about how BERT works,” *ArXiv200212327 Cs*, Nov. 2020, Accessed: Apr. 29, 2022. [Online]. Available: http://arxiv.org/abs/2002.12327

[18] T. Yako and 八箇毅, “Insert molding of plate glass fitted product,”
JPH06246782A, Sep. 06, 1994 Accessed: Apr. 29, 2022. [Online]. Available:
https://patents.google.com/patent/JPH06246782A/en

[19] A. Akbik, T. Bergmann, D. Blythe, K. Rasul, S. Schweter, and R. Vollgraf, “FLAIR: An Easy-to-Use Framework for State-of-the-Art NLP,” in *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)*, Minneapolis, Minnesota, Jun. 2019, pp. 54–59. doi: 10.18653/v1/N19-4010.

[20] I. Hendrickx et al., “SemEval-2010 task 8: multi-way classification of semantic relations between pairs of nominals,” in *Proceedings of the Workshop on Semantic Evaluations: Recent Achievements and Future Directions - DEW ’09*, Boulder, Colorado, 2009, p. 94. doi: 10.3115/1621969.1621986.

[21] Y. Liu et al., “RoBERTa: A Robustly Optimized BERT Pretraining Approach,” *ArXiv190711692 Cs*, Jul. 2019, Accessed: Apr. 30, 2022. [Online]. Available: http://arxiv.org/abs/1907.11692