Separating Precision from Recall, and the Use of
Machine Learning Methods for the Generation ofHigh Value Patent Landscape Collections
Managing Director, Patinformatics, LLC

Anthony Trippe

Anthony（Tony）Trippe is Managing Director of Patinformatics, LLC. Patinformatics is anadvisory firm specializing in patent analytics and landscaping to support decision making fortechnology based businesses. In addition to operating Patinformatics, Mr. Trippe is also an
Adjunct Professor of IP Management and Markets at Illinois Institute of Technology teaching acourse on patent analysis, and landscapes for strategic decision making. Tony is an influential
thought leader in the patent analysis space, recently wrote the Guidelines for the Preparationof Patent Landscape Reports for the World Intellectual Property Organization, has been
named one of the Top 300 IP Strategists by IAM Magazine, and is the author of the popular
Patinformatics blog.

Measuring Patent Information
Retrieval Effectiveness

1 INTRODUCTION2

It can be argued that the single most important

In the data science world, information retrieval,

step in generating a patent landscape reportor searching effectiveness is traditionally（PLR）is the creation of an appropriate data

described in terms of two measures, recall andcollection for performing the corresponding

precision. These items are defined as:

analysis. The consequences of creating a suboptimal data collection goes by many names,⃝Recall – how much of the useful information

but most often the acronym used to describe

has my search retrieved?

this situation is GIGO — “garbage in, garbage⃝Precision – how much of the information thatout”. In the case of patent landscape reports

I have retrieved is useful?

the data most frequently used is collectedby performing a patent search. The patentStated another way, recall is an estimate of

information retrieval must be fit for purpose, or

the probability that a relevant document will be

the analyst runs the risk of producing results

retrieved in response to a query and precisionthat are irrelevant at best, and incorrect andis an estimate of the probability that a retrieved

misleading at worst. These poor results are thedocument will be relevant. Figure 1 provides agarbage out that GIGO refers to. Understanding

demonstration of these concepts1:

the consequences of building a sub-standard

Thinking about the issues in searching during

patent collection for the generation of a PLR,

the preparation of a PLR, information retrieval

how does an analyst measure the quality of amethods usually look at precision and recall

patent search, and perhaps more importantly

simultaneously and measure their methods

how do they optimize their approach to building

by how search techniques stack up against

a collection that will produce the most relevant

both elements. Even though this is the case,results.

166

1

https://www.tutorialspoint.com/data_mining/dm_
mining_text_data.htm

Stated another way, recall is an estimate of the probability that a relevant document will be
retrieved in response to a query and precision is an estimate of the probability that a retrieved
document will be relevant. Figure 1 provides a demonstration of these concepts1:

Figure 1: A Venn Diagram of Recall
vs.1:Precision
FigureA Venn Diagram of Recall vs. Precision

recall there is usually a subsequent drop in the

Coming back to the topic of high precisionlevel of precision.
Generally speaking, as PLR
1keep in mind that when statistical analysis ishttps://www.tutorialspoint.com/data_mining/dm_mining_text_data.htm

searches are often designed to maximize recall,performed on large, or macro-level sets only

the precision of the results can suffer sincemajor trends, or items that appear frequently

more off-topic references may get included inare going to be seen. Precision, in this instance,

the collection. This can produce the “garbage

can suffer to some degree with these types ofin” scenario described above if something

searches, since minor occurrences within theseisnʼt done to also improve the precision of thesets will not be seen in the larger context. This

collection.

can often be evaluated by examining several of
the significant trends to ensure that they are

Having said this though, it is also important tocoming from reasonably precise references. If

keep in mind that there is a point of diminishing

this is the case then it is generally acceptable

returns when it comes to high precision. Highto sacrifice some precision for the sake of recall.

recall is always desirable since it is important

From personal experience a greater than 80%

to have as much of the relevant information

precision is required to avoid the GIGO scenario.

available for analysis as possible when buildinga PLR. A good rule of thumb is for analysts

As previously stated though, in order to get

to try and achieve at least 90% recall into 90% recall there is a high likelihood that

their collection. One way to measure this isprecision will suffer, and could in fact fall below

to intentionally broaden a search by adding

the 80% threshold. Since this is the case how

additional search terms, or classification codes

does an analyst get to the point where they canto the query. If the majority of the new records

achieve the desired thresholds with regards todiscovered were not in the previous iteration ofboth measures. In generating collections forthe search, and most of the new records are not

PLRs, it might be more productive to begin by

relevant then the analyst can begin to feel thatcreating sets using methods that produce highthe search has achieved the desired level ofrecall, and then look to increase the precision

recall.

separately once the initial, high recall collection
is built. This approach is contrary to traditional
information retrieval methods that try to maximize
both simultaneously.

YEAR BOOK 2O17 167

2

特許情報の高度な活用

Separating Precision and Recall
precision and recall are normally opposed tofor Patent
RetrievalThinking about the issues in searching during 3
the preparationof Informationa PLR, information
retrieval
one another such that with an increase inmethods usually look at precision and recall simultaneously and measure their methods by how

4 Maximizing Recall

below are several search methods that are used
to query patent collections:Of the two tasks associated with optimizing
recall, and precision, maximizing recall is by⃝Reasonably defined keyword hedges —far the easiest. Almost anyone can search

including the use of synonyms and proximitywith a couple of intentionally broad patent

operators as well as broader Boolean operatorsclassification codes by simply truncating⃝Narrow patent classification codes — atthem to four-digits. For instance, searching

the group/subgroup/class/subclass levelwith A01H （New Plants or Processes forsome codes are specific enough that they

Obtaining Them; Plant Reproduction by Tissue

produce precise results by themselves

Culture Techniques）, or C12N（Biochemistry⃝Citation analysis — backward citations in— Microorganisms or Enzymes）will generate aparticular are often used to identify relevant

pretty comprehensive collection of documents

references

associated with plant breeding. The issue in this⃝Semantic analysis — using techniques suchcase is that this particular search will generate

as natural language processing relevant

more than 2 million worldwide records. Keeping

records can be found that weren’t retrievedin mind that no competent patent searcher

using keywords and Boolean

would ever suggest building a query like this it
is still illustrative of what could be done to try to

To further improve recall these methods canachieve nearly absolute recall. This would be thebe used in combination with one another, forultimate example of “garbage in, garbage out”

instance:

since much of the output associated with ananalysis of this collection would probably havevery little to do with plant breeding, assuming
it would even be possible to produce results
based on the analysis of this many records.

⃝Broader patent classification codes can bequalified with broader keyword hedges⃝Citation, and Semantic analyses can befiltered by using broad patent classificationcodes

In a similar fashion, an obscenely broad
search can be accomplished using keywords,

All of these methods often produce additional

for example, a search of plant “and” breeding inresults beyond the ones generated when thethe full-text of worldwide patents would produce

more specific versions of them alone were used.

over 300,000 records. While this is certainly

When all of these approaches are combined aless than over 2 million this approach suffers

high recall collection is normally produced, andfrom similar issues as the query that involved

can be verified by comparing the difference

overly broad use of patent classification codes.

between these results with an overly broad
search and checking to see if any of the unique

Perhaps more constructively, one of themore useful ways of increasing recall without
producing collections that are overwhelmingly
broad is to use multiple searches that incorporate
a variety of search tools. For example, listed

168

references associated with the overly broad
search are in fact relevant.

5

Maximizing Precision with Machine
Learning Methods“Machine learning, a branch of Statistical
Learning, is about the construction and study ofsystems that can learn from data. For example,

a high recall, but reasonably sized collection

a machine learning system could be trained on

there is still a likelihood that the patent records

email messages to learn to distinguish between

will not meet the 80% precision threshold thatspam and non-spam messages. After learning,is desirable for a high-quality landscape data

it can then be used to classify new email

set. Many of these collections will also be quite

messages into spam and non-spam folders.”

large, often numbering in the thousands, ortens of thousands of patent families, so the

There are many machine learning methods thattraditional method of manually reviewing evencan be applied to patent information retrieval,

titles, or enhanced titles for relevance would be

and analytics including text clustering, and aa large, tedious chore. So now that recall has

more involved interpretation of clustering tobeen maximized how does an analyst approachproduce spatial concept maps. For the purposes

the issue of precision, especially with largerof improving precision though, classification isdata collections?

the method that is likely to be of the highest
value. Classification is usually accomplished

One potential solution for increasing precision

with a supervised, machine learning method thatin a patent data set is to turn to the field ofuses “learning sets” to identify key attributes

machine learning for organizing, and prioritizingof documents in a category. The “learning sets”

documents. These methods have quickly

are small sub-collections, one for each category,

become some of the most polarizing tasks

generated by the analyst, who decides whichassociated with patent analytics. While

test documents should appear in each class.

these approaches have caught on, and are

New documents are compared to the learning

used in many industries, the adoption in thecollections, and assigned to a class based

patent information space has been sporadic.

on their similarity to the documents that have

Having said this, the tide may be turning withalready been assigned to the category.

regards to implementation of machine learning
methods for patent information retrieval. At

When it comes to classification the mostthe most recent PIUG Annual Meeting held in

frequently applied supervised machine learning

May of 2017 a significant percentage of themethods are Artificial Neural Networks（ANNs）,

presentations covered machine learning, andand Support Vector Machines（SVMs）:

2

many of the vendors introduced products thatincorporated it in one form, or another. So what

⃝Artificial Neural Networks 4 – In computer

exactly is machine learning?

science and related fields, artificial neuralnetworks are models inspired by animal

Consulting Wikipedia the following definition iscentral nervous systems（in particular thefound :
3

brain）that are capable of machine learningand pattern recognition. They are usually2

https://www.piug.org/an17program3

https://en.wikipedia.org/wiki/Machine_learning4

https://en.wikipedia.org/wiki/Artificial_neural_network

YEAR BOOK 2O17 1692

特許情報の高度な活用

Even using the methods above to producepresented as systems of interconnected

method6:“neurons” that can compute values from
inputs by feeding information through the
network.

In machine learning and statistics, classificationis the problem of identifying to which of a set of– supervised

categories（sub-populations）a new observation

learning models with associated learning

belongs, on the basis of a training set of data

algorithms that analyze data and recognize

containing observations（or instances）whose

patterns, used for classification and regression

category membership is known.

⃝Support Vector Machines

5

analysis. The basic SVM takes a set of
input data and predicts, for each given input,

An algorithm that implements classification,

which of two possible classes forms theespecially in a concrete implementation, is

output, making it a non-probabilistic binary

known as a classifier. The term “classifier”

linear classifier. Given a set of training

sometimes also refers to the mathematicalexamples, each marked as belonging to one

function, implemented by a classificationof two categories; an SVM training algorithm

algorithm, that maps input data to a category.

builds a model that assigns new examples
into one category or the other.

6

Binary Classification Using a Support
Vector Machine to Increase Precision

Classification can be thought of as two
separate problems – binary classification andmulticlass classification. In binary classification,a better understood task, only two classes

The methods used for automatic classification

are involved, whereas multiclass classification

have been around for some time, and have

involves assigning an object to one of several

been used by patent offices, publishers andclasses. Since many classification methods

database producers, in association with patent

have been developed specifically for binary

information, but there have not been many

classification, multiclass classification often

commercial tools providing classification

requires the combined use of multiple binary

capabilities to analysts, and patent information

classifiers.

retrieval specialists. This is changing however,170

and in the not too distant future patent information

When it comes to increasing precision in a

professionals will have a variety of machine

patent data collection binary classification is thelearning based classification systems available

method that should be used. For this article, ato them. Before launching into an example

support vector machine（SVM）implementationof how classification can assist with theof binary classification will be used for the task.

identification, and prioritization of relevant

Referring back to the Wikipedia reference onreferences, within large patent document sets,

SVMs the motivation behind the method, and an

let’s look at some details of the task itself.

illustration are provided:

The Wikipedia entry on Statistical Classification

Suppose some given data points eachprovides the following description of thisbelong to one of two classes, and the goal is56https://en.wikipedia.org/wiki/Support_vector_machinehttps://en.wikipedia.org/wiki/Statistical_classification

to decide which class a new data point will be

An Example, Putting these
Ideas into Practice7

data point is viewed as a p-dimensional vector

Several years ago, the author developed（a list of p items）, and we want to know whether

an interest in wearable fitness monitors andwe can separate such points with a（p − 1）began using this field as an example when-dimensional hyperplane. This is called a

exploring machine learning methods and thelinear classifier. There are many hyperplanes

problem of recall, and precision in patentthat might classify the data. One reasonable

data collections. Two of the major companies

working in the space at the time were Aliphcom
choice as the best hyperplane is the one that
Suppose some given data points each belong to one of two classes, and the goal is to decide（doing abusiness
represents
thelargestorofmargin,which
class a new
data
point willseparation,
be in. In the case
support vector machines,
data point isas Jawbone）and Nike. Bothviewed as a p‐dimensional vector (a list of p items), and we want to knowwhether
we cansell other products, and haveorganizations
between the two classes.
separate such points with a (p − 1)‐dimensional hyperplane. This is called a linear classifier.
extensive
patentThere are many hyperplanes that might classify the data. One reasonablechoice as thebest portfolios, which cover their
hyperplane is the one that represents the largest separation, or margin, between the twofitness monitors, as well as many additional
A theoretical example of this can be seen inclasses.
items. A binary classifier, using a SVM can help
Figure 2:
A theoretical example of this can be seen in Figure 2:
identify the patents associated with personal
fitness monitors in the midst of many otherpatents from these companies.

Searching worldwide, several hundred patentdocuments are assigned to Aliphcom. Of
these, more than 100 are associated with theirpersonal fitness band, based on a previous
analysis conducted using a manual method ofclassification. Ten of these documents were used
to represent the positive examples in the training
Figure 2: Illustration of separating hyperplanes in Support Vector Machines

Figure 2: Illustration of separating hyperplanes in Support
Vector Machinesset. The Aliphcom portfolio also contains patentdocuments associated with Bluetooth headsets
Binary classification provides a means for categorizing large collections of patent documents
and speakers. Ten documents associated with
into the references that are likely to be of highest interest to the information professional, andthosethat arelikely not related, provides
but were stillaretrieved
a high recallthesesearch. items
The training
set, identified as the negative
wereBinaryclassificationmeansin forin this case will be made up of references that would be found in a high precision version of theexamples.
categorizing
largethecollectionsof patent
documents
records that are
not relevant also needcollection.
In trainingclassifier, examplesof patent
to be identified, so the classifier can establish a hyperplane that will distinguish between theinto the references that are likely to be of highest
two categories.
The first step in preparing to build a classifierinterest to the information professional, andVII.
An Example, Putting these Ideas into Practiceismonitors
to decideon the sections of text that will bethose years
thatago,arethelikelyrelated,but were
still fitness
Severalauthornot
developed
an interestin wearable
and began
using
this
field
as
an
examplewhen
exploringmachinelearning
methods
andtheproblemof individual document vectors.
used to create theretrieved in a high recall search. The training set,recall, and precision in patent data collections. Two of the major companies working in the
In Boththisorganizations
case, thesell
source titles, and abstracts
space
at thetimewill
werebeAliphcom
as Jawbone)in thiscase
made (doing
up ofbusinessreferencesthat and Nike.
other products, and have extensive patent portfolios, which cover their fitness monitors, aswere used, but in other circumstances enhanced
would be found in a high precision version of thecollection. In training the classifier, examples of

titles, and abstracts, or source claims could

patent records that are not relevant also needbe used as well. All potential family members

to be identified, so the classifier can establish awere imported in this case, but frankly, under

hyperplane that will distinguish between the two

normal usage, it’ s probably a good idea to put

categories.

the documents through some type of family

YEAR BOOK 2O17 1712

特許情報の高度な活用

in. In the case of support vector machines, areduction before performing a classification

generated from the classification effort ontask. The source titles and abstracts can be

the Aliphcom patents. The initial use of theused for a binary classification since the user

Aliphcom classifier did not produce particularly

is simply trying to separate relevant documents

good results. This was to be expected sincefrom the remainder of the collection.

the language used in the Aliphcom records wasdifferent from the text used in the Nike portfolio.

The next step involves training the classifier,

Ultimately, after additional training, a classifier

and as discussed above, ten positive and ten

will need to be able to handle this eventuality

negative examples were initially chosen tosince patent collections are rarely homogeneous

accomplish this. Depending on the application

with regards to language, especially between

used to generate the SVM there will be various

different companies. Having looked at patents

methods for selecting the documents for the

associated with the Nike personal fitnesstraining set. Once the initial classification ismonitor using traditional review methods, many

accomplished most systems will present theof the known documents did not score well withanalyst with a set of at least ten middle-of-the-the Aliphcom training data. This situation wasroad documents that the classifier is having

remedied by retraining the classifier, as wasdifficulty classifying. The analyst is asked todone with the first Aliphcom classifier. Afterlook at these, and manually classify them before

three generations of training, the classifier hada second training round takes place with thesuccessful scored ~85% of the Nike documents

new training set.

accurately. It still scored some of the originallydiscovered documents poorly, but frankly, many

After only three training rounds a classifierof these were associated more with a sensor

was created that successfully classified all

system that was similar but distinctly different

but one of the Aliphcom documents correctly

than the personal fitness monitor. Interestingly,

into those covering the personal fitness monitor

the classifier in this case identified several

compared with the remainder of the companyʼs

Nike families that were not discovered using

products. The one document, and its equivalent

the original, traditional search. Combing thefamily members were new, recently published,methods, in this case, would have led to a moreand dealt with a new application of the productcomprehensive result when studying the Nikeline. All and all, with minimal effort, a result

fitness monitor filings.

with greater than 95% precision was achieved.
In this particular example, working with a few

Finally, and representing an even bigger

hundred records where about 25% are relevant

challenge for a computer assisted method of

to start with is a fairly simple test. A realclassification 43,612 US, and WO documents

challenge would be whether a SVM classifierin International Patent Classification class

could be used to classify larger, more highly

A61B005, the class under which the majority

diversified portfolios.

of the relevant documents analyzed to thispoint were assigned, were classified using the

To t e st t hi s , 1 1 , 1 2 6 w o rl d wi d e p at ent

Aliphcom and Nike classifiers.

documents from Nike were submitted to the
SVM for classification using the final classifier

172

This is an enormous number of extraordinarily

of whether the patent information professional

machine learning method. A61B005 is the IPC

is interested in Patentability, Freedom-to-class for measuring for diagnostic purposes,

Operate, or Validity it is often the case thatand it includes MRI, and blood glucose monitoring

results are presented in reverse chronologic

as well as the fitness devices being investigated.

order, as opposed to relevancy order. Even when

The language used in the titles and abstracts,

a relevancy rank is provided it is sometimesof these documents, can be very different

difficult to determine what criteria was usedthan what was used in the Nike and Aliphcom

to generate the ranking. With the use of adocuments.

SVM the analyst is in control of choosing thetraining set, evaluating the results, and deciding

The first attempt at classification produced

how many retraining sessions are required tovery poor results with a few documents

generate a sub-collection which is highly likely

receiving a high score and a handful thatto be related to source patents of interest. In

received reasonable scores, but were clearly off-

these circumstances, a classifier could save

topic. Following the previous pattern, retraining

significant time in reviewing patent records.

of the initial model was performed. Due to thesize and diversity of the collection, this process

Information retrieval systems typically look

was repeated five times before a reasonableat precision and recall simultaneously andoutcome was produced. Using the fifth-generation

measure their results by how techniques stackclassifier, 620 documents received a score ofup against both elements. When it comes to50, or better. The titles of these were studied,patent searching it might be more productive

and from a preliminary examination it appeared

to separate these functions so that theyas if ~80% of these were on topic.

can be maximized independently. It has been
demonstrated that precision and recall do not

8 Conclusions

follow the same linear path when producing a

Taking a large number of documents frombe more productive to begin with methods thata diverse IPC collection is the ultimate test

produce high recall exclusive of precision. Oncefor a machine learning method, and in this

this is accomplished the results can be ranked

example the SVM performed reasonably. In

using different methods to improve precision

real world situations, it is recommended thatand manage the way the results are sharedfamily equivalents be removed and documentwith the analyst. It will likely be the case thatcollections created that are not quite so broad.

different approaches will be used to provide

Alternatively, additional retraining sessions will

higher recall than those that can be employed

help bring more relevant references over theto share records with higher precision. Instead

relevance threshold.

of expecting a single method to do both it would

patent search. Since this is the case it might

be useful to the patent searching, and analysis
While patent landscape searches were thefocus of this particular article the machinecommunity if the process was done stepwise to
maximize the value to the analyst.

learning methods discussed can also be applied
to other patent searches as well. Regardless

YEAR BOOK 2O17 1732

特許情報の高度な活用

diverse documents, and a very tall order for a高付加価値特許ランドスケープ収集を作成するための適合率／再現率の分離と機械学習法の利用（抄録）Anthony Trippe
１．はじめに

4．再現率を最大化する

特許ランドスケープ報告書（PLR）を作成する際の重必要な分類コードを使えば、ある事例では、200 万

要事項はその分析に相応しい特許集合を準備することで件が直ぐに集められ再現率も高い。まさに、“garbage

ある。最適化の前の集合状態を示す頭字語は GIGO（ゴin, garbage out” である。全文検索によりキーワード

ミ入れ、ゴミ出し）と呼ぶ。その収集は検索によって集キーだけで検索すると 30 万件がヒットする。200 万められる。その収集は目的に相応しいものでなければな

件よりかなり少ないので良くない。再現率を高く保ちならない。さもないと、分析結果がせいぜい不適切で、最がら適合率を下げない工夫として各種の検索ツールを組悪の場合にはミスリードを犯すリスクが伴う。このようみ合わせるのが良い。その例を下記に示す。

な不適切な結果をゴミ出しと言う。PLR を産む准最適⃝広義のブーリアン演算だけでなく同義語を含む近接演化された収集を構築するためには、その特許検索の品質を如何に測るかが重要だ。

算を利用する。
⃝適切な狭い特許分類コードを使う。
⃝後方引用を含む引用被引用分析を利用する。

2．特許情報検索の有効性の測定
データ科学では、伝統的に再現率と適合率の 2 つの⃝キーワードやブーリアン検索で得られなかった自然言
語処理の概念検索を利用する。

測定因子で測られてきた。それぞれの定義と説明は、図⃝広義の特許分類を広義のキーワードで組み合わせる。

1 を参照されたい。一般的には、再現率を上げるとそ⃝引用および概念分析の結果を広義の特許分類を使用しの結果として適合率が下がってくる。一般的にいっててフィルタリングを行う。

PLR は再現率を最大にしようとするから garbage inシナリオになる。良い例は、再現率が少なくとも 90%

5．機械学習法を用いて適合率を最大化する

以上の時に履行される。だから検索用語を追加し、分類

Wikipedia には各種の用語が定義されている。

コードを追加することが多い。個人的な経験から判断す

1）機械学習ると、GIGO シナリオを回避するには適合率 80% 以上

2）人工ニューラルネットワークが求められる。

3）サポートベクターマシン（SVM）
4）統計的分類

3．特 許情報検索のために再現率と適合率を5）SVM による二値化識別

分離する
重要な語句が頻繁に出現する大量データを扱う場合に

6．適 合率を増加させるための SVM を利用

は再現率が犠牲になっても傾向が解るが、重要な語句のした二値化識別（重要な主張ポイント）

頻度が低い場合には要注意である。経験から言うと適合二値化識別は大量の特許データを最も関心が高く可率が 80 以上では GIGO シナリオに陥るのを避けるこ

能性が高い特許文献と、検索はされたが関連性が高くなとが必須である。再現率を 90％以上にすると再現率がい可能性がある特許文献に識別する手段を提供する。二悪くなるのが一般的である。良い PLR を得るには先ずつのカテゴリーに区別する識別超平面を描くことができ

高い再現率を求め、次に適合率を高めたい。しかし、こる。（図 2 参照）（回帰問題として解く。）の方法は伝統的な再現率と適合率を同時に高めようというやり方とは反対のやり方である。

7．当アイデアを実践する事例研究
数年前にウェアラブルフィットネスモニターへの関心が高まり、機械学習の方法と、再現率の問題と特許

174

データ収集の適合率の問題を検討した。Aliphcom 社とさせ、分析者と結果を共有する方法を管理して、結果をNike 社の 2 社のデータを使って SVM を利用して二値ランク付けすることができる。再現率と適合率を同時に化識別を行った事例が役立った。

高める方法を期待するのでなく、分離して検討し、分析最初に、Aliphcom の少数の事例のタイトル、要約、者の価値を最大化するために段階的プロセスを行うこと

請求項を対象にして識別を行った。次に、良い該当事例だ。それは、特許検索や分析コミュニティにも有益なもを 10 個抽出し、該当から外れた事例を 10 個抽出した。

のである。

この最初の方の処理段階で二値化識別が難しく、中間領2

特許情報の高度な活用

域に混在する場合も発生するが、その場合にはトレーニ
ングをやり直した。この二値化識別のトレーニングを三回も繰り返すと、適合率が 95％より大きくするこができるようになった。
そこで、Aliphcom 社の教師データを使って、Nike社の 11,126 件の特許のカテゴリー化を試みた。三回のトレーニングの後には、85％程度の識別成功率がえられた。
別 の A61B005 分 類 の 43,612 件 の US 特 許 と
WO 文献でも検討を行った。三回のトレーニングでも識別がうまくできない場合には 5 回目の見直しを行う
五世代識別を行った結果、Score が 50 以上の 620
件の特許文献を抽出することができた。

8．結論
ここで議論された機械学習をこの記事の焦点の特許ランドスケープ検索に応用した。他の特許検索にも適用が可能である。特許専門家が特許性調査、FTO 調査、特許有効性調査に興味を持つか持たないかに関係なく、この手法の結果は時系列とは関係なく提示されることが多い。関連性ランクが提供されている場合でも、ランク付
けを生成するためにどの基準が使用されたのかが解らない場合が多い。SVM を利用することにより、分析者
はトレーニングセットの選択、その結果の評価、および関心あるソース特許に関連する可能性が高いサブコレク
ションの生成に必要な再トレーニングセッションの数を決めなければならない。大量の特許文献のレビューをする際には識別法はかなりの時間節約になる。
特許情報検索システムは、通常、適合率と再現率の両方を同時に探求する。しかし、特許検索ではそれらが独立して稼働させるために分離して扱う方がより生産的である場合もあることを実証している。だから、先ず高い再現率を生成する手法から始める方が生産的である。これを達成してから、異なる方法を活用して適合率を向上

YEAR BOOK 2O17 175