A study on improving accuracy to predict
confidential words in judicial precedents using
the neural network
Masakazu Kanazawa1 , Atsushi Ito1 , Kazuyuki Yamasawa2 , and Takehiko
Kasahara3
1

Utsunomiya University, 7-1-2 Yoto, Utsunomiya-shi, Tochigi, 105-0123 ,Japankanama2591@gmail.com, at.ito@is.utsunomiya-u.ac.jp
2
TKC corporation, Karukozaka-MN Bldg. 2-1
Ageba-Cho,Shinjuku-ku,Tokyo,162-0824 ,Japanyamasawa-kazuyuki@tkc.co.jp
3Toin Yokohama University, 1614 Kuroganecyo, Aoba-ku, Yokohama-shi, Kanagawa,
225-8503 ,Japankasahara@toin.ac.jp

Abstract. ICT (information and communications technology) has been
adopted in the judicial fields of multiple countries. In Japan, the courts
do not open all judicial precedents to the public due to privacy issues.
In available precedents, confidential words, e.g., personal, corporate, andplace names, are manually replaced by a single letter. This replacement
process takes time and eﬀort; therefore, not all precedents are opened.
To solve this problem, we introduce a technique to automatically predict
confidential words using a neural network. In a previous study, we demonstrated that our method is eﬀective for predicting confidential words.
However, the ac-curacy of the prediction was not suﬃcient for practicaluse. Therefore, we attempted to improve the detection accuracy for confidential words. First, we updated the parameters of the proposed model,such as the window size. Specifically, we enlarged the window size; thisled to a better accuracy when detecting target words; however, the prediction accuracy for the confidential words decreased. In addition, prior
to the learning in the neural network, we modified the input dataset
to easily predict the target words. Unfortunately, using this method,
we could not obtain good accuracy. Therefore, it is likely that the taskof predicting the confidential words might be diﬃcult when only using
a neural network. One possibility to improve the accuracy is to use aproper noun dictionary. We explain this idea in detail in this paper.
Keywords: cyber court · judicial precedent · confidential word · neuralnetwork · proper noun dictionary

1

Introduction

Because the business world is increasingly recognizing the Internet as a place
where commercial dealings can take place, the legal system needs to be equippedto not only understand the Internet but also to use it as a tool to successfully
exchange and use information. The Internet now provides a wide range of legal
information, and one of the benefits of information being provided in this way
is that it can be kept up-to-date as the law changes. Not only can the Internet
assist in legal research but it can also assist in court processes in general, that is,in trial preparation and in the courtroom throughout the hearing. Reasons thatcourts should embrace such technology lie in the constantly increasing caseloads,
the complexity of cases and jurisdictions, resource constraints, the pressure to
improve access to justice, expectations of improvements in performance, and thepressure to improve eﬃciency and eﬀectiveness in the court s administration andthe delivery of justice [1]. Therefore, jurisdictions of many countries aim to follow
economic growth and political changes to allow anyone to easily access judiciary
documents by introducing ICT (information and communications technology).
Such a justice system empowered by ICT is called a cyber court . The pioneer
study of a cyber court system is Courtroom 21 [2]; which started in 1993 at theCollege of William & Mary as a joint project between the university and the
National Center for State Courts in the U.S.A. They implemented a cyber court
using a teleconference system and performed multiple trials to determine theeﬀectiveness of ICT in the court. In addition, the Singaporean Supreme Court
introduced ICT in 1998 via measures such as a centralized display management
system, a digital transcription system, e-signatures, electronic hearings, and an
online justice system. All documents are displayed on PC screens in the court [3].
The High Court of Delhi and the District Court of Delhi have also introduced
paperless e-courts. From the standpoint of technological research, there are manystudies focusing on models combining law and the knowledge structures of law[4, 5]. Even China began stream some trials in more traditional courtrooms online in 2016 in an apparent eﬀort to boost the transparency of their legal system[6]. Accordingly, we decided that it is very important to develop a prototype of acyber court to evaluate the eﬀectiveness of such an idea. In Japan, the prototype
for the first civil trial was developed at the Toin University of Yokohama in 2004
[7, 8], and its eﬀectiveness, particularly its usefulness in the Saibanin system(the Japanese jury system), was proven [9]. An experiment with a remote trial
was also conducted [10]. The Investments for the Future Strategy 2017 by theJapanese Cabinet Oﬃce includes ICT conversion for trials to accelerate them andimproving their eﬃciency [11]. From the viewpoint of using big data, this field is
expected to develop further. The disclosure of judicial precedents is indispensable for using big data in AI and in the field of law; however, most precedents are
not publicly open on Japanese court webpages. This is because Japanese privacy
stipulates that an individual not be specified. Online privacy in Japan is primarily governed by a general law, the Act on the Protection of Personal Information(APPI), rather than a specialized law for online privacy. The APPI applies toall business operators that possess individuals personal information. Japan hasother personal information protection laws that apply to government and publicorganizations. Confidential words (e.g., personal, corporate, and place names) in
opened precedents are replaced by other words, such as a single uppercase letter.

This operation takes time and eﬀort because it is done manually. In addition,
because trials involving people from various countries have been increasing due
to globalization in recent years, creating a proper noun dictionary is diﬃcult.
However, neural networks have advanced greatly in the field of natural language
processing in recent years. Studies including deriving a vector considering themeaning of words and predicting words are actively ongoing [12]. Therefore, we
proposed two neural network models: a bi-directional LSTM (Long Short-term
Memory) LR (left right) model and a Sum-LSTM based on the CBOW (continuous bag-of-words) model. We experimented with these models to ascertain
their eﬀectiveness for predicting confidential words. As a result, we were able
to predict confidential words but did not obtain a good accuracy. In this paper,
we review our experimental results and then we experiment with the proposed
neural network by changing the window size and choosing proper parameter values. Then, we enhance the preprocessing of the input datasets. This operation
is important for the neural network learning process. We evaluate the results ofthe experiment and propose a new model combined with a proper noun dictionary to form a more powerful method to improve the accuracy. In Section 2, we
explain our initial experimental results. In Section 3, we discuss the result of anexperiment with a modified window size. In Section 4, we discuss using a proper
noun dictionary to increase the accuracy for predicting confidential words. In
Section 5, we summarize our study and discuss future avenues of research.

2

Review of our comparative experiment on predicting
the confidential words in Japanese precedents

2.1

A comparative experiment of four prediction method and theexperimentJapanese precedents include many confidential words such as personal names,
corporate names, and place names. Due to the Japanese understanding of privacy, such words are converted into other words to protect privacy. In Japan,
this procedure takes time and eﬀort because it is done manually. Some judicial
precedents are available on the website of the court [13]. Confidential words inthese precedents have been replaced with an uppercase letter of the alphabet.
(In paid magazines and websites, Japanese letters are sometimes used.) Figure
1 shows an example of a replaced word. This process is performed manually. To
do these process automatically this processes, we proposed the following neuralnetwork models shown in Fig. 2. We experimented with the two proposed models to determine the prediction eﬀect for confidential words. The two models
are the bi-directional LSTM-LR model, which has been shown to be eﬀective
for modeling and predicting sequential data (see Fig. 2(b)) and the Sum-LSTM
model, which is based on CBOW, which can represent the similarity of a word(see Fig. 2(c)). In this experiment, we also tested a simple LSTM model (seeFig. 2(a)) and a Sum bi-directional LSTM-LR, which combined the Sum-LSTM
and LSTM-LR models (see Fig. 2(d)), to compare to the two proposed models.

Fig. 1. Example of a replaced word

Fig. 2. The four neural network models used in the first experiment

2.2

Analyzing the result of the experimentThis experiment was conducted using the four models shown in Figure 2. Also,
the embedding vector was 200 for all models. The judicial precedents used inthis experiment are 20,000 precedents on the Japanese court website. Various
parameters are shown in Table 1. For accuracy, we used perplexity (PPL) thatwas used in a previous research for predicting the next word. PPL was given by
the following equation:
1
(1)
PPL =
P (correctword)
In Eq. (1), P indicates the probability and PPL represents the number ofprediction choices that are narrowed down to neural networks. The smaller thevalue, the better the prediction results. CW PPL is the average PPL of thetest data and reflects the perplexity of the confidential words, while PPL isthe average for all the test data. The PPL scores are shown in Table 2. The
result indicates that the two proposed models were both eﬀective with respectTable 1. Parameters of the model

Embedding size 200
Window size 10
Batch size200

to the PPL in our corpus. However, when we look at the results of CW PPL,
which is directly related to the task of predicting the confidential words, wefind that the diﬀerence in the scores was at least 32.687 between PPL and
CW PPL in the case of the bi-directional LSTM-LR model. This result suggeststhat the task of predicting confidential words is more diﬃcult. The PPL scoresindicate that all the proposed methods are more eﬀective than the simple LSTM
model. However, for the prediction of confidential words, only the bi-directional
LSTM-LR model showed good results. Sum-LSTM based on CBOW might haveproduced these results. CBOW is an eﬀective model for paraphrasing words,
and Sum-LSTM uses this mechanism. Therefore, when Sum-LSTM predicted
a word that was confidential , the CW PPL score became worse due to thepossibility of paraphrasing words such as plaintiﬀ , defendant , doctor , andteacher . Knowing the paraphrased versions of the confidential words meantthat embedding vectors of the confidential words could be successfully generated
and that the model could recognize the meaning of confidential . However, theprediction accuracy did not improve; therefore, it is possible that there might beproblems when calculating the probability of the prediction task. In summary,the first experiment showed that our proposed models using neural networks
were eﬀective for predicting confidential words; nevertheless, the accuracy ofthese models need to be improved [14].

3Changing the window size

3.1

Problems with the first experiment

We had the following problems in the first experiment for the bi-directional
LSTM-LR model, which scored the best score compared to the other models.
(1) The PPL score was poor for continuous numeric words. Continuous numericwords scored approximately ten times worse than the target words. It is likelythat our neural network model had a weakness for the continuous numeric words
to train the many test dataset involved numeric words.
(2) The CW PPL score was poor due to the possibility of paraphrasing words
such as plaintiﬀ , defendant , doctor , and teacher . To solve thisproblem, these para-phrasing words could be excluded from the choices whencalculating the probability. In addition, it is important to examine scores otherthan PPL. These models were based on a paper by Mikolov [15]. However, there
are other important parameters.
3.2

The aim of the experiment and the plan to improve theaccuracy

Our ultimate goal is to obtain high accuracy when automatically predicting
confidential words. Therefore, we solved the problem described in Section 3.1
and upgraded the detection accuracy for confidential words using a neural network. We designed an experiment based on the first experiment, that is, thebi-directional LSTM-LR model (Fig. 3). First, we review the parameters, suchas the window size, of our neural network model to obtain the proper value forthe window size. In addition, prior to the learning step in our proposed neuralnetwork, to predict the target words more easily, we enhance the input dataset
via preprocessing by changing to a powerful morphological analyzer. Simultaneously, we avoid unnecessary words, i.e., stop words , in the input dataset.
Finally, in the next section, we propose a method to improve the accuracy forpredicting confidential words. In this method, we combine a proper noun dictionary with our proposed neural network.

Fig. 3. the experiment using the model (Bi-directional LSTM LR)

Fig. 4. Values of PPL and CW PPL for diﬀerent window sizes

3.2.1 Changing the window size
First, we focused on the window size. In general, a neural network such as LSTM
has a chunk size (which we call the window size in this paper). The chunk size isthe number of (output) frames for each chunk of data evaluated in the training or
decoding. RNN or LSTM models, or chain models, are always trained on fairly
large chunks (generally in the range of 40 ‒ 150 frames). During decoding, we
also generally evaluate the neural network using fairly large chunks of data (e.g.,30, 50, or 100 frames); this is usually referred to as the frames-per-chunk. For
recurrent networks, the chunk-size/frames-per-chunk and the extra-left-context
and extra-right-context are approximately the same during training and decoding because this generally gives the best results (even though sometimes it is thebest to make the extra-context values slightly larger in decoding). One might
expect that in decoding time longer context would always be better; however,this does not always seem to be the case [16]. If the length of a chunk is too
short, the association between chunks cannot be learned, which will aﬀect theaccuracy of the LSTM model [17]. In our first experiment, we used a window size
at 10 frames (words), as shown in Table 1. At that time, we did know if that thisnumber was appropriate. Accordingly, we changed the window size parameter from 10 to 20 because we expected the PPL score to be improved by larger
training samples. We expected to obtain the correct results [18]. In addition, wechanged the window size from 10 to 5 to investigate how the training ability is
aﬀected by narrowing the window size. We see that the PPL values significantly
worsen in this case. The PPL scores are seven times larger than those in theoriginal experiment, and the CW PPL values further worsened. Therefore, thewindow size greatly influences the accuracy when detecting confidential words.
Next, we enlarged the window size from 10 to 20 or 30 and trained the neuralnetwork. The resulting PPL and CW PPL scores are shown in Fig. 4. The PPL
and CW PPL scores are best when the window size is 20. When we enlarge thewindow size to 30, the PPL score is good but the CW PPL score is worse. The
CW PPL score is likely poor when the window size is too large because, in thiscase, a large number of confidential words are contained in the window andthe neural network has many choices. We also changed the value of epoch .
When training a neural network, one epoch indicates one pass through the fulltraining set. Usually, a few iterations are used. We experimented with epoch
values of 1, 5, and 100. Normally, when the epoch value is larger, the predicting
accuracy is better. However, if the epoch value is too large, over-training will
occur and the accuracy will decrease. In our experiment, even when the epoch
value was 1, PPL changed little while CW PPL worsened. We assume that anepoch value of 100 is better for predicting confidential words. The result is shownin Fig. 4.
3.2.2 Enhancement of the input dataset processing prior to learning(1)Update of the morphological analyzer ”MeCab” to ”MeCab-ipadicNEologd”

Fig. 5. Data preprocessing step

Fig. 6. MeCab upgrade

In the first experiment, we used 50,000 judicial precedents for the training
data and 10,000 judicial precedents for the test data. These data included therecords of trials from 1993 to 2017. We used the precedent database provided by
TKC, a Japanese corporation [19]. Before these datasets flow into our proposed
neural network, we converted all the confidential words contained in the datasetsto the uppercase letter A and separated the Japanese words with spaces using MeCab , a Japanese morphological analyzer [20], because we were using
the Japanese judicial precedent dataset. These processes were performed in thedata-preprocessing step (Fig. 5). In the second experiment, we decided to use
MeCab-ipadic-NEologd , which is a more powerful morphological analyzer,
has better performance than MeCab , and is an open software. We expected
the prediction accuracy to improve for the confidential words (Fig. 6). However,
the result of the experiment shows that the accuracy decreased compared to thefirst experiment (Table 3).
That is, there was no change in the PPL value but the CW PPL value became

two times worse. It is possible that, because the ability of the morphologicalanalysis increased, overlearning occurred. In addition, the neural network parameters changed. Therefore, we need to continue the experiment to obtain theproper parameters, e.g., the epoch size and the window size.
(2) Legal terminology stop words
Because we are dealing with judicial words in the corpus, legal terminologies
frequently appear in the corpus. These words do not necessary predict confidential words. Therefore, it is important to exclude these legal terminologies fromthe corpus. Accordingly, we registered these judicial words in a stop words list.
Then, we removed these words from the corpus. The evaluation of the results
will require further study.

4

Using a dictionary

4.1

Our new proposals to improve accuracy using the proper noun
dictionary with our neural network model

The first experiment demonstrated that the task of predicting confidential words
with high accuracy is diﬃcult. We decided to improve the accuracy via a newmodel using a proper noun dictionary as input to the neural network. If thetarget word appears in the dictionary, it is easy to provide the correct answer.
Even if the target word does not appear in the dictionary, if it is a confidential
word, the input vectors will have a natural pattern. Therefore, we expect toachieve high accuracy if we combine a dictionary with the prediction method.
We display the flow of the current process, and that for future development, inFig. 7.

Fig. 7. Outline of this paper

4.2

Existing method of extracting the proper noun

Conventional proper noun extraction is performed using a named entity dictionary. The named entity extraction API extracts a named entity such as aperson’s name, a place name, or a date representation from the Japanese character string sent in the request. Then, we manually search for confidential words.
These named entity words are useful for detecting confidential words. We provide an overview of this method in Fig. 8. This method eventually obtains theconfidential words manually.
4.3

Our proposed method combined with a proper noun dictionary

4.3.1 One method using a proper noun dictionary
Confidential words in Japanese judicial words are usually a pronoun, such as
personal name or a place name. We focus on this point. Accordingly, we use aproper noun dictionary with our neural network. There are several ways of using
the dictionary. One method is to match the input corpus and the dictionary
and then input the resulting data into the neural network (Fig. 9). Because thismethod is processed step by step, it is time intensive.
4.3.2 Using a proper noun dictionary
Conversely, our proposed method is processed directly because we combine thedictionary with the neural network. We describe the algorithm of our proposedmodel as follows. We use a proper noun dictionary, that is, the well-known
morphological analyzer MeCab-ipadic-Neologd (MeCab-N), combined with
our proposed neural network. When the corpus is input into MeCab-N, MeCab-N

Fig. 8. Existing method of extracting the proper noun

Fig. 9. using proper noun dictionary

disassembles the information into parts that contain a noun, verb, and adjective.
Then, we choose the noun, e.g., the place name or the personal name, fromthe output data disassembled by MeCab-N. Next, we assign a unique index tothe part information and make a part dictionary, merging them into the input
data (word) for our neural network model via an embedding vector (Fig. 10).By
using the proper noun dictionary information as input to the neural network, it
is expected that the accuracy will increase. If the target word is in the proper
noun dictionary, it is easy to identify. Even if the target word is not in the proper
noun dictionary, if it is a confidential word, the input vectors are the same as
the easy to recognize pattern. Therefore, we expect to obtain high accuracy.
However, parameter adjustments for our proposed neural network are importantbecause there have been no prior studies on such a model.

5Conclusions

Privacy is one of the most important issues in cyber courts. However, it is noteasy to completely hide confidential words such as personal information and locations. One technique to protect privacy is to find confidential words in a fileor on a website and convert them into meaningless words. It would be useful
to have a neural network that is intelligent enough to automatically anonymize

Fig. 10. Marging the proper noun dictionary with the neural network

confidential words. Based on Japanese judicial precedents, we have already proposed a recognition technique for confidential words using a neural network. In
this paper, we proposed measures to improve the model s prediction accuracy
for confidential words. The first measure was to adjust the parameters of the
proposed model to increase the accuracy. When we changed the window sizeof our proposed neural network to be larger and when we adjusted the epoch
value, the PPL score was acceptable; however, the CW PPL score was poor.
This is because, in our method of replacing the confidential word, it is diﬃcult
to obtain the confidential word; therefore, we need to review and change our prediction method. The second measure was to improve the preprocessing. Prior tolearning the input corpus, we enhanced the preprocessing of the corpus using theadvanced analyzer MeCab-ipadic-NEologd ; we hoped that this would improve
the CW PPL score; however, this was not the case for both PPL and CW PPL.
It is likely that a parameter adjustment is necessary. Finally, we proposed a newmodel that combines a proper noun dictionary with the neural network. Thismodel has the ability to predict target words using our proposed neural network
as well as the ability to match target words, such as confidential words, using aproper noun dictionary. If this model works well, we will establish an automatic
detector tool for confidential words.

Acknowledgments We express special thanks to all people who joined the
Cyber Court Project.

References
1. www7.scu.edu.au/1878/com1878.htm
2. http://law.wm.edu/academics/intellectuallife/researchcenters/clct/ [Sep, 5, 2017]
3. https://www.supremecourt.gov.sg/default.aspx?pgID=361No,10 [Sep, 5, 2017]
4. JURISIN, Japan, Oct 2013
5. JURISIN, Japan, Oct 2014

6. https://www.theverge.com/tech/2017/8/18/16167836/china-cyber-courthangzhou-internet-disputes
7. Takehiko Kasahara: Cybercourt - Court Technology: Journal of the Japanese Society for Artificial Intelligence, vol. 23, No. 4, pp 513-, Jul. 2008]
8. Takehiko Kasahara: Court Technology for Civil Procedure: Festschrift for the seventieth birthday of Prof. Takeshi Kojima Ⅱ, pp. 961-, Sep. 2009
9. Canyoujudge?:NHK
Special,2005.2.13,
PM9:00:
http://www6.nhk.or.jp/special/detail/index.html?aid=20050213 [Sep, 5, 2017]
10. A study of applying ICT for legal service: Report of a research funded by the
Minis-try of Ministry of Internal Aﬀairs and Communications in Japan in 2009,(March 2010)
11. http://www5.cao.go.jp/keizai-shimon/kaigi/minutes/2017/0609/shiryo 07.pdf
12. LAI, Siwei, et al. How to generate a good word embedding. IEEE Intelligent
Systems, 2016, 31.6: 5-14 http://www.courts.go.jp/app/hanrei jp/search1 [Sep, 5,
2017]
13. http://www.courts.go.jp/app/hanrei jp/search1 [Sep, 5, 2017] Yuya Kiryu, Atsushi Ito, and Masakazu Kanazawa: Acta Polytechnica Hungarica [Sep9 2018]
14. Yuya Kiryu, Atsushi Ito, and Masakazu Kanazawa: Acta Polytechnica Hungarica[Sep9 2018]15. MIKOLOV, Tomas, et al. : Eﬃcient estimation of word representations in vector space: arXiv preprint arXiv:1301.3781, 2013. http://www.tkc.jp/law/lawlibrary[Sep 6, 2017]16. http://kaldi-asr.org/doc/dnn3 scripts context.html
17. LSTM-Based Hierarchical Denoising Network for Android Malware Detection (Security and Communication Networks Vol. 2018, Article ID 5249190, 18 pages)
18. Masakazu KANAZAWA, Atsushi ITO, Yuya KIRYU, et al.: Improve accuracy of
predicting confidential words in judicial precedents: JURISIN, Japan, Oct 2014
Technical Committee on Thought and Language (TL) [July,28,2018] LSTM-BasedHierarchical Denoising Network for Android Malware Detection (Security andCommunication Networks Vol. 2018, Article ID 5249190, 18 pages)
19. http://www.tkc.jp/law/lawlibrary [Sep 6, 2017]20. http://taku910.github.io/mecab/ [Mar 9, 2018]