DEIM Forum 2023 1a-1-2

Label-dependent Paragraph Selection forEﬃcient Neural Long Document Classiﬁcation
Tan-Ning HUANG† and Makoto P. KATO†† University of Tsukuba
1-2 Kasuga, Tsukuba, Ibaraki 305–8550, Japan
E-mail: †s2126087@s.tsukuba.ac.jp, ††mpkato@acm.org
Abstract BERT is incapable of eﬀectively processing long documents due to its quadratically-growth self-attention mechanism. A common solution can be truncating documents under BERT’s maximum input length, but
it will be considered to give negative eﬀects on classiﬁcation performance. To address this problem, we propose
several approaches let BERT capable to handle long documents. We ﬁrst extract paragraphs by matching phrases
identical or associated with the target label, therefore truncating a 512-word paragraph representing the original full-length document. In extend of our truncating operation, we introduce a machine-learning based paragraph scoringmechanism. By predicting the classiﬁcation conﬁdence probability of several paragraph-equivalents, our methods
are able to rate the eﬀectiveness between each paragraph-equivalents by a regression analysis model. Aggregatedfrom a Japanese novel self-publishing website as our task dataset, we deploy several classiﬁcation tasks to compare
these approaches with baselines. Experimental results suggested our approaches show a notable improvement in
performance if under a 512-word limitation. However, non-neural network models with full-length document input
have the highest performance up to this point.
Key words Long document classiﬁcation, Application of natural language processing, Language model

1 Introduction

ing model which successfully applied to various tasks, including document classiﬁcation. By its self-attention mech-

The fast-growing information-based society not only gives

anism, BERT is able to eﬀectively apprehend the context

us more opportunities to fulﬁll our needs but also overﬁlled

and meaning of words. This is critical to document-level

us with irrelevant noises. In 2022, Japan has published overclassiﬁcation since it requires the classiﬁer to understand

68,000 books across multiple domains [1]. On the other hand,the context between phrases. [5] Another beneﬁt to utiliz-a Japanese will only read around 15 16 books annually on

ing BERT on document classiﬁcation tasks can be praised

average [2], which is 0.022% by published books. Therefore,

as its availability to eﬀectively handle long-range dependen-

it is important for readers to ﬁnd the proper book that theycies, where BERT can understand the relationships between

are willing to ﬂip through. Nonetheless, modern public ac-phrases that are sparse in the document. This is impor-cess catalog systems commonly need users who already knowtant because the purpose of a document depends on thewhat they are looking for and are barely available to suggest

relationships between phrases throughout the entire docu-

topic-related items. These kinds of systems are generally dis-ment. [6] However, by BERT’s transformer architecture andtinct from humans’ search behavior, thus we need to heavily

self-attention mechanism, it suﬀers in a O(n2 ) complexity

rely on document ranking methods for recommending un-

in memory and time, which quadratically grows with the

known titles. Document classiﬁcation by natural language

length of the input sequence. [7] In general, the maximum

processing has become a familiar approach to accompany in-

input length of BERT models is about several hundred to-formation retrieval systems. It has been proven by several

kens, such that the base version of BERT can only process

domains such as legal [3] or biology domain [4]. These kinds

its input sequences up to 512 tokens. [6] This makes BERTof documents are often found within large corpus with com-models unable to eﬀectively handle long documents.

plex contexts, making them challenging from conventional
statistical models.

To address this issue, we propose our approach for adapting BERT to operate on long documents, this involves trun-

Bidirectional Encoder Representations from Transform-

cating long documents by a 512-word paragraph to meet theers, BERT, is a state-of-the-art natural language process-maximum input limitation of BERT. However, if we truncatethe documents without considering their context will lead toclassiﬁcation accuracy. However, at present, the FULL doc-

a decline in classiﬁcation performance. [8] As a result, it isument still can perform the highest classiﬁcation accuracy.

necessary for us to conﬁrm the truncated paragraphs are able

In this paper, we present and compare various document

to preserve the authentic context of the document. Therefore

truncating methods for Japanese ﬁction literature docu-

we identiﬁed three requirements for our proposed method:ments. Our speciﬁc contributions include:(1) Truncating the document to a maximum of 512 words;（ 1 ） In order to avoid BERT’s maximum 512-token in-(2) Preserving the original context of the document; (3) An

put limitation, we proposed several truncating methods toeﬃcient truncating operation. It is important to note that

handle long documents.

we do not trim the documents at the token level in order to（ 2 ） To improve the accuracy of document classiﬁcation

avoid using the vectorizer during the truncating operation.

tasks, we apply a machine-learning strategy to distinguish

Our approach is based on the hypothesis that, if a para-eﬀective paragraphs.

graph containing lexicons identical or similar to the classi-

In Section 2, we will discuss related work and compare theﬁcation target, should keep the same context as the orig-

similarities and diﬀerences between our approaches. Our ap-

inal document. Therefore, we can reduce the document’s

proaches will be presented in Section 3. Section 4 will discuss

contents by the section having similar lexicons.

our experiment details and ﬁndings.

For im-plementation, we propose two basic truncating operations:First-Match(FM) and Nearest-K (NK). FM traverses

2 Related works

the document from the beginning and searches for phrases

Since its publication in 2018, Bidirectional Encoder Repre-

identical to the target label. A 512-word paragraph is thensentations from Transformers (BERT) has become a state-of-

selected after the identiﬁed phrase. However, FM is unable

the-art model and has been applied to multiple NLP tasks,

to apply to all documents due to the searching phrases need-including document classiﬁcation. Based on its multi-layer

ing to be equivalent to the target lexicon. To address thisTransformer encoder, BERT enables a pre-training mecha-

limitation, we then propose the Nearest-K (NK) trunca-

nism that allows the language model eﬀectively learn fromtion operation. Based on Word2Vec, NK forming a similar-a large-scale corpus before ﬁne-tuning for a more speciﬁc

vocabulary list and identifying the phrases that appear in thetask. [10] During this process, the number of input tokens

document. The size of the list is represented by variable K.

for BERT is typically limited to a ﬁxed number, usually 512

The NK method will then truncates the paragraph that has

tokens. [6] Since BERT’s attention mechanism does not only

the most occurrences of these matched similar phrases.

focus on ﬁxed positions or special tokens but broadly ref-In addition to our hypothesis-based approaches, we merge

erences the whole sentences in lower layers. [11] This makes

our truncating operations with machine-learning approaches.

BERT challenging to train on longer documents due to the

We introduced Conﬁdence-Learning (CL) method, which

suﬀering from a O(n2 ) time complexity. Several approaches

is a bottom-up instance-based approach. The CL method

working on this attention mechanism of BERT have been

involves tagging multiple truncated paragraph-equivalents

conceived to allow longer documents exceeding 512 tokens,with the conﬁdence-score ranked by a ﬁne-tuned language

therefore being able to process by BERT-like models. Long-

model. Therefore, by training a regression model, we canformer [7] and Big bird [12] introduced the sliding window at-

predict each paragraph’s possible eﬀectiveness for the classi-

tention mechanism, where instead of full-length global atten-

ﬁcation task.

tion, Longformer and Big bird allow local-focused attention

Our long documents dataset was collected from Kakuy-

pattern that is available for linear complexity. Both Long-omu (Hatena, co ltd., kakuyomu.com), and consists of 69,627

former and Big bird can process input sequences with up to

documents with an average length of 127,000 words across4,096 tokens, which is suﬃcient for most classiﬁcation tasks,

40 labels. We conduct our experiment using DocBERT [9],but they still hold a ﬁxed-length input limitation which is aa BERT variant for document-level classiﬁcation that also

drawback for processing overly-long documents.

shares a maximum input length of 512 tokens. To verify

A common approach to handling these kinds of docu-

our proposed truncating operations, we include two baseline

ments without modifying the language model is to divide

operations: First-512 (F512) and FULL document. Thethem into smaller segments.: Kong et al. [13] and Khandve

F512 method involves simply truncating paragraphs fromet al. [14] proposed approaches that divided documents into

the beginning at 512 words, while the FULL method con-

smaller segments that could be processed by BERT individ-

tains the entire document without any truncation. Our ex-ually. With statistics or neural networks, they managed toperimental result shows that, when operating under the 512-

coordinate classiﬁcation results based on output aggregatedword limitation, our proposed methods signiﬁcantly improvefrom segments. Another similar approach presented by He etal. [15] combined segmentation with a recurrent neural net-eral, the goal of our research is to ﬁnd a truncating operation

work. Their model only focuses on signiﬁcant phrases rather

that maximizes classiﬁcation accuracy, therefore, is eﬀective

than the whole document, allowing for more eﬃcient pro-for document classiﬁcation tasks.

cessing of the input. The encoding produced by the model’s
local attention mechanism is then processed by a recurrent
neural network decoder in a sequential manner. Although

3. 2 Hypothesis
To address the research question, we proposed our experiment methods based on two hypotheses:

segmentation is eﬀective at handling long documents, its disadvantages can be recognized in that the model still needs
to process the entire document, resulting in a high time cost

H1:
If the truncated paragraph contains the same phrase as totarget label, the contextual information remains intact.

in total when handling extremely long documents.
BERT is pre-trained on general text corpora, such as
BookCorpus [16] and Wikipedia. This makes BERT ideal
for cross-domain language processing tasks but lacking taskspeciﬁc or domain-related knowledge. [17] This makes BERT

H2:
If we accept phrases whose lexical is similar to the target
label, it is also as eﬀective as the phrase we use in H1.

ineﬃcient on some speciﬁc tasks which rely on domainrelated knowledge. Legal documents, diﬀer from the corpora

Therefore we proposed two truncation operations, First-

BERT was originally pre-trained on, are often exceed the

Match (FM) and Nearest-K (NK) to test our hypothe-maximum input sequence length limitation. Limsopathamses. We also tried to combine our truncating operation withet al. [8] utilized diﬀerent approaches, including simple trun-a machine-learning strategy to make advancements in our

cation, segmentation, and the use of Logformer and Big bird,

proposed methods:

to apply BERT-based model to the legal domain.

Theyfound that Longformer and Big bird can achieve better per-H3:

formance on long legal document classiﬁcation tasks, while

If we label truncated paragraphs with conﬁdence-scores

simply truncating or discarding part of the document with-generated by a ﬁne-tuned BERT model, we can determine

out regarding the document’s context can result in unsatis-the expression of which paragraph will have higher classiﬁ-

factory performance. In general, utilizing language models

cation performance by a machine-learning approach.

that are speciﬁcally designed to handle longer documents canbe a practical solution for handling long document classiﬁca-

Here we deﬁne the conﬁdence-score as ”The possibil-tion tasks. However, this does not fully resolve the problem

ity of correctly classifying the document evaluate byof maximum input length, as the input is still limited to aa ﬁne-tuned BERT model.” Therefore, the conﬁdence-

particular extent.

score should be a serial number from 0.0 to 1.0, for us to

3 Methodology

conduct a regression model to train on the relation between
conﬁdence-scores and truncated paragraphs.

We imple-

3. 1 Reseatch Question

mented this approach as the Conﬁdence-Learning (CL)

As we mentioned before, BERT can not eﬀectively han-method.

dle long documents that exceed its maximum input sequence

3. 3 Deﬁnition

length of 512 tokens. Consequently, we need to truncate long

Based on our hypothesis, we can deﬁne our research ques-

documents under BERT’s input limitation, at a maximum oftion as, by giving any long document d and target label l,

512 tokens, to apply BERT in the long-document classiﬁca-

we are able to ﬁnd a truncating operation Trunc(d, l) thattion task. Regardless, truncating the document into a shorterproduces a shortened paragraph p(d,l) of the original docu-

version without any consideration will result in poorer per-ment, which is a sub-string of d, the truncating operation isformance due to the loss of contextual information. [8]diﬀerent by each proposed operation, we will caption them

Therefore, our research question can be expressed as:
RQ:
How can contextual information of the long doc-in later subsections.
p(d,l) = Trunc(d, l) = d[start:end]

ument be preserved while truncating into a shorter

Where d[start:end] is a sub-string of d beginning from theparagraph?

positional index start while closing at index end.

As our intent, we need to ﬁnd a solution to obtain a para-

As our research question suggests, each truncated para-graph that not only fulﬁlls the 512 tokens input limitationgraph p should obtain the contextual information of the orig-

but also keeps the contextual information unharmed. In gen-inal document. The contextual information is deﬁned as ”Ifthe target label can be recognized in the original

apply FULL document for classiﬁcation, consequently, thedocument.” Thus, the contextual information should be

information loss of this method should be minimal. However,a boolean variable indicating the condition that if the para-

the extra-long document length will cause extensive compu-

graph, or document, is labeled by target label. This suggests

tation costs, where the input length of FULL document isthat the truncated paragraph p(d,l) should contain the same

unbearable for most of the neural network-based language

classiﬁcation result as how the original document is labeled,models.

implies that the classiﬁcation task Cls(p, l) is a binary clas-

3. 4. 2 First-Match (FM)Based on our ﬁrst hypothesis, the FM method traverses

siﬁcation task:{
Cls(p, l) =1

if f (p, l) > 0.50otherwisef (p, l) = sigmoid(W · BERTl (p)[CLS] )3. 4 Truncating operations
As the previous section suggested, blindly truncating documents by head or tail is been proven to have poorer eﬀectiveness on the classiﬁcation tasks due to the loss of contextual information. [8] Hence, our truncating operations shouldmeet the following criteria:（ 1 ） The truncating operation can be applied to all doc-

the document from beginning to end searching for the ﬁrst
phrase that is identical to the target label. We then truncated a 512-word paragraph after the matching phrase. If
no phrase is matched in the document, we select F512 paragraph instead. First, we introduce Wl , which is a set ofphrases representing target label l, which is manually aggregated. For example, the Wl for the target label ”超能力
(id:42)” consists of phrases including ”超能力”, ”能力”, ”異能力”, ”異能”, etc. Therefore if any of these phrases in thislist is matched, we consider it as matching the target label.
Accordingly, we are able to deﬁne truncating operation FM
as follows:

uments.

Truncf m (d, l) = d[fm:fm+j]（ 2 ） The truncating operation should keep the contex-

tual information of the original document unchanged.

Where fm is the positional index generated by the matching（ 3 ） The truncating operation should be eﬃcient andprocess:

able to avoid high computation costs.

fm = min{i|d[m:m+n] ∈ Wl }In the next subsection, we will discuss our baseline truncat-

This method had moderate complexity, while the number ofing operation, followed by our proposed methods. At last,

application documents can vary by the target label since thewe will introduce a machine-learning aggregated approach.

phrase in the document must be identical to the target label

3. 4. 1 Baseline
In our baseline truncating operation, we consider two
methods: Front512 (F512) and FULL. Represent themindless truncating operation and the ideal scenario if notlist as a requirement.
Example for a ParagraphTruncated by FM with Label ID 96（戦争）:

impossible. By the F512 truncating operation, we simply

戦争は終わらずにずっと続いていたのだった。ようや

truncate any phrases by the ﬁrst 512 words that appeared

く目的地に着いた。風にはためく軍旗が誇らしげに掲

at the beginning of the document. Therefore, we deﬁne our

げられ、扉の両脇は長槍を持った軍服姿のケット・シー

F512 operation as:が固めている。旗の意匠は氷の結晶に渦巻く吹雪と交差する槍。リスティンキーラ軍司令部だ。俺が帰還す

Truncf 512 (d, l) = d[1:j]Here, the positional index start and end are ﬁxed numbers
for representing the 1st and 512th words of the document.
The value of j is varied by each document but will always
locate at the ﬁnal character of the 512th word. F512 isbelieved to be the easiest and least complex operation with
O(1), however, it is considered to have the most tremendous
information loss during the operation.
On other hand, in FULL scenario, we use the full-length
document without any truncating operation.
Truncf ull (d, l) = d[1:|d|]ることは通達済みなため、特にトラブルもなく入口を通過。目的地である三階を目指して階段を登っ…
デウス・エクス・マキナを殺せ, 2020, ほりえる [18]There are several variations of the FM operation: LastMatch (LM) used the last matched vocabulary for truncation. FM-Front-and-Back (FM-FNB) truncated theparagraph before and after the matching phrase, respectively.
3. 4. 3 Nearest-K(NK)To address the problem that FM may have led to a deﬁciency of applicable documents, we proposed our second
approach with the Nearest-K operation. In short, we compared the similarity between the target label and each vocab-

In practice, we do not need to make any modiﬁcations toulary appearing in the document, then truncated a 512-wordparagraph having the most related phrases. To be more spe-

3. 4. 4 Conﬁdence-Learning(CL)ciﬁc, based on each target label and its corresponding sub-In previous sections, we propose FM and NK, two ba-

collection dataset, we broke up the vocabulary in all doc-

sic truncating operations based on the hypothesis that theuments. Behind that, we used word2vec to calculate thecontextual information may be preserved by having identi-

relation score between the target label and each phrase to

cal or similar phrases to the target label. In this section,

create a list of similar vocabularies. We then selected thewe would like to discuss another truncating operation based

top-K similar vocabularies by the size of K and comparedon a machine-learning approach.

each phrase within a document to ﬁnd the paragraph whereparagraph selection method aggregating truncating opera-

the top-K vocabularies appeared the most. In this exper-tion and machine learning. We ﬁrst truncate documents byiment, we only built a similar vocabulary list using nouns

paragraph alternatives, diﬀerent paragraphs truncated fromsince they tend to show the most signiﬁcant diﬀerences by

the same document, as basic. For example, the paragrapheach label.

alternatives of FM, FM(top-10) will be the 10 paragraphs

CL, an instance-based

We calculate the similarity between words w that appearedin which the matching phrase ﬁrst appears. N-100(top-in the document to target label l by the cosine similarity,10) will be the ﬁrst 10 paragraphs which applied to N-100,

therefore:ranked by score from high to low.
By having multiple paragraphs truncated from the sameew · el∥ew ∥ ∥el ∥
1 ∑
el =
ew
|Wl | w∈Wsim(w, l) =document, in an ideal situation, every alternative paragraphshould have the same context as the original document. However, in practice, the result by the classiﬁer might vary due tol(k)Hence, we are able to acquire the similar word set Sl

the variation between each paragraph does not have the sameasthe set of k-words, corresponding to Wl , ranked by similarity.
By these premises, we deﬁned the NK operation as:
Truncnk (d, l) = argmaxp∈Pd∑context. Therefore, if we were able to comprehend the expression between good paragraphs and bad paragraphs ,we should be able to distinguish which expression from aparagraph is suﬃcient to document classiﬁcation tasks.

tf (p, w)

To be more speciﬁc, we trained our CL application usingw∈Slk

these procedures:p ∈ Pd = {d[i:i+j] |1 <=i<
= |d| − j}（ 1 ） We obtained paragraph alternatives by a truncating

The NK approach is a more complex operation, since notoperation. (FM, NK, etc.)

only did NK is needed to create a similar vocabulary list,（ 2 ） With a ﬁne-tuned BERT classiﬁer, we embedded thebut it also requires applying K words to each document for

conﬁdence-score with each alternative.

the matching process. However, NK had the beneﬁt that（ 3 ） Combining the conﬁdence-score with paragraph al-almost all of the document could be found with aternatives, we trained a regression model to predict therelated512-word paragraph if K size is large enough. In general,
N-100 (K = 100) provided a good balance between timerequirements and application documents.
Example for a ParagraphTruncated by N-100 with Label ID 204（勇者）:conﬁdence-score.
The conﬁdence-score is the possibility of correctly classifying the document by a ﬁne-tuned BERT model. For example,
if we have a document that has the contextual informationof the target label id 42 (l42 ), therefore by having two para-Sl100= { 勇者、魔王、戦士、英雄、賢者、ラスボス…}
42graph alternatives truncated by N-100: p0 as the 1st result

不安な表情を浮かべる少年に、アリコが自己紹介をすthat these two paragraphs should have the same contextualる。「申し遅れました。私は王国騎士隊の曹長、アリinformation Cls(p0 , l42 ) = Cls(p1 , l42 ). Regardless, if we

コと申します。あなた様のお名前は？」「……僕はユウ

apply a ﬁne-tuned BERT classiﬁer, we may ﬁnd out thatトです」──この後、ユウトはアリマン王、フリル大

f (p0 , l42 ) = 0.97 and f (p1 , l42 ) = 0.51, indicates that BERT臣と謁見した。伝説の勇者であるか？を確認するため

has higher conﬁdence to classify p0 than p1 as label l42 . Asにユウトは勇者の装備を装着したが、勇者の装備を装

result, we can assume that the expression of p0 is superior

着して軽々と動くユウト…

to p1 , therefore having a higher conﬁdence score.

伝説の勇者様！
？ ……いえ、補助職の僧侶です,

2020, 田山照巳 [19]return from Truncnk and p1 as the 2nd, while can assume

Therefore, we deﬁne our CL operation as:

Trunccl (d, l) = argmax(Regressionl (p))
p∈P dp ∈ Pd = {d[start:end] |Trunc(d, l)}In our experiment, we generated 10 truncated paragraphalternatives using the N-100 method, ranked by the number of similar vocabulary words. If a document did not haveat least 10 alternatives, we randomly selected a 512-wordparagraph until we retained a total of 10. We then used aﬁne-tuned BERT model trained with FM-FNB for the language model by step (2). In step (3), we utilized a Rigid
Regression model and Logistic Regression models to train
our paragraph alternatives by target conﬁdence-scores.

4 Experiment
In the previous section, we addressed the research problemof adapting BERT for long document classiﬁcation. Here, we
present our approaches to addressing the problem: we proposed two truncating operations for creating shortened paragraphs from the original document, with the aim of improving classiﬁcation accuracy compared to baseline approaches.
In this section, we would ﬁrst provide an overview of thedataset used in this study (Section 4.1). Therefore We will
describe the general experiment setup (Section 4.2), followedby the ﬁndings from our proposed approaches (Section 4.3).
4. 1 Dataset
The diﬀerence between literature ﬁction documents andothers is obvious, literature ﬁction has more literary context,
is usually based on imagination, and is more emotional. [20]
It is also important to note that literary ﬁction is usuallymuch longer than other documents. For example, news articles can range from hundreds to thousands of words [21],online review comments are typically around 300 [22], while
longer documents such as legal-domain documents [8] andacademic papers [14] are around 10,000 words. On the other
hand, literary ﬁction can vary in length depending on thegenre or author but is generally around 20,000 to 40,000
words for a short story and 120,000 to 180,000 words for along story. [23]To address the unique characteristics of literature ﬁction
documents, we designed our dataset based on Kakuyomu(Hatena, co ltd., kakuyomu.com), an online novel publishing service that allows users to publish their personal works.
There are several beneﬁts to using ﬁction works gatheredfrom Kakuyomu as our dataset: (1) all works are publishedopenly and cover a wide range of genres and characteristics; (2) most of the works are labeled and classiﬁed in detailby the authors; (3) the length of each work is considerably
lengthy.
We acquired 69,627 ﬁction works with a total of 237 la-bels, each work by length of 127,000 words on average. We
then excluded low-priority labels for which the number ofsuggested works was less than 1,000, resulting in a dataset
containing 40 labels with its own sub-collection. To create abalanced sub-collection, we selected all ﬁction works with thetarget label, as well as an equal number of randomly chosen
works from other sub-collections that were not classiﬁed by
the target label. We then partitioned each sub-collection into
three training sets in an 8:1:1 ratio for training, validation,and testing, respectively.
4. 2 Experiment setup
The aim of our experiment is to create a document classiﬁcation task using a 512-word paragraph dataset as training data. This dataset was based on the original full-length
dataset and is organized according to the label. In general,our document classiﬁcation task was dependent on each label, with the classiﬁcation target being a binary outcome.
As we were training for a binary classiﬁcation task, the language model’s predictions were limited to two classes: Trueor False. Consequently, the values of the evaluation indices(precision, accuracy, f1 score) were the same. In this paper,
we would use accuracy for the evaluation index.
More speciﬁcally, our experiment consists of four steps:（ 1 ） Extracting a sub-collection dataset based on the tar-

get label.
（ 2 ） Truncating each document in the sub-collection us-

ing our truncation method.
（ 3 ） Training the truncated sub-collection with a lan-

guage model.
（ 4 ） Verifying the accuracy of the training results.

In our experiment, we applied several language models including neural network models such as DocBERT [9], KimCNN [24], HAN [25] and Reg-LSTM [26], as well as statisticalmodels like linear regression, support vector machine, andnaive Bayes. However, we primarily used DocBERT as themain model for our experiment.
4. 3 Experimental results
The results of our approaches were shown in Table 1. The
vertical axis represents the truncation method applied to the
documents, while the horizontal axis represents the language
model used for training. The values in each cell represent the
average accuracy across all 40 labels. As previously mentioned, the FULL documents are too long to be processedby neural network models, so there are no values for thiscondition in Table 1.
Our results indicate that the BERT model consistently
outperforms the other language models in terms of accuracy.
However, there is not a clear diﬀerence in performance between neural network models and statistical models, except
for BERT and naive Bayes. When considering only para-

Table 1 Results for Baseline, FM and NK
BERT

Kim-CNN

HAN

Reg-LSTM

LR

SVM

F512

0.7162

0.66260.68750.68060.69200.6839B-NaiveBayes M-NaiveBayes
0.68280.6776

FULL

-

-

-

-

0.75930.76830.6406

0.6799FM

0.73650.70640.72220.71350.71680.71150.7096

0.6991FM-FNB

0.74570.70130.71510.71700.71610.712

0.70810.6979LM

0.74000.70720.72090.70810.70840.70550.70130.6956

LM-FNB

0.74300.70290.71910.70850.71090.70710.70010.6995N-100.74040.70020.71300.70240.72010.71550.71280.6928N-20

0.73820.70160.71700.71460.72060.72010.71250.6956N-30

0.73930.70060.70740.72140.72040.71480.71280.6964N-100

0.74550.70170.70850.71280.72390.72050.71240.7005graphs with a maximum of 512 words, our proposed meth-H3:

ods all have higher performance than the baseline F512, al-If we label truncated paragraphs with conﬁdence-scores

though the methods using FULL document input still havegenerated by a ﬁne-tuned BERT model, we can determine

the highest accuracy, even without the BERT model. Whenthe expression of which paragraph will have higher classiﬁ-

comparing the diﬀerent truncation methods, there is no sig-

cation performance by a machine-learning approach.

niﬁcant diﬀerence in performance between them However,
FM-FNB and N-100 do show slightly better results.

Table 2 Results for CL

Based on the results of our experiment, we can draw the

Ridge

LR

Ideal

0.74650.74320.9349following conclusions:
N-100(top-10)

H1:
If the truncated paragraph contains the same phrase as totarget label, the contextual information remains intact.

The results for CL are presented in Table 2. All of thelanguage models used in this experiment were trained with
the N-100 method for 10 paragraph alternatives. The val-

Our hypothesis, H1 is conﬁrmed. Since all of the FM vari-

ues in each cell represent the accuracy of the CL methods

ants can perform a better classiﬁcation accuracy comparedin selecting the correct paragraph based on the highest pre-to baseline F512 in all language models, suggesting that

dicted conﬁdence-score. An ideal situation is also included

including phrases equivalent to the target label can give ain the table to show whether any of the 10 variants can bepositive eﬀect on training the language model. However, thecorrectly classiﬁed.

result that FULL document input is able to outperform ourAs result, we can ﬁnd that CL can correctly select overtruncating operation, implies that we still experience some

74.6% of eﬀective paragraphs which can be correctively clas-

loss of contextual information or noise from negative sam-siﬁed by the BERT model, however, the result can onlyples.

slightly overcome FM-FNB, therefore, hypothesis H3 is partially conﬁrmed. Since we cannot ensure that the improve-

H2:
If we accept phrases whose lexical is similar to the target
label, it is also as eﬀective as the phrase we use in H1.

ment of CL was not resulted by irregularity.

5 Conclusion and Future WorkIn this work, we addressed the problem that BERT could

We can partially conﬁrm hypothesis (H2) by our experi-

not eﬀectively process long documents that exceeded 512 to-ment result. Just as FM, NK show a similar result whenkens due to its self-attention mechanism. Either we would

compared to baselines, implying that similar phrases can beexperience prolonged processing times by the lengthy doc-

equivalent to the target label phrases we use in FM opera-ument, or degraded performance if the content is truncatedtion. However, we cannot verify the distinction through FM

indiscriminately. To address this problem, we proposed threeand NK, with only a slight accuracy diﬀerence, thus it is nottruncating operations to convert lengthy documents into 512-

clear whether using similar vocabulary can actually have anword paragraphs while preserving the contextual informa-

impact on classiﬁcation accuracy since phrases identical totion of the original documents. We ﬁrst introduced the FM

the target label is also included in the similar phrases list.

method, which truncated the paragraph by identifying theﬁrst occurrence of the target label within the document. Asa follow-up, we proposed the NK method, which utilized
the word2vec to identify the similar vocabularies scattered[12]

in the document, then truncated the paragraph accordingly.
Finally, we proposed the CL method based on a machinelearning approach to identify superior paragraphs that are
more eﬀective on document classiﬁcation tasks. Our dataset[13]

was sourced from Kakuyomu, a Japanese self-publishing website for novels, known for its collection of extra-long docu-[14]

ments that oﬀer rich context. We then conducted experiments to evaluate the eﬀectiveness of our proposed truncating operations in comparison to the baselines. Our results
suggested that our methods can eﬀectively extract relevant
context from the original document while adhering to the[15]

512-word limitation. However, our approaches could not surpass the performance of the baseline that used the FULL
document as input. For future work, we planned to fur-

[16]

ther investigate the eﬀectiveness of our approaches with additional datasets and other language models. We also intend
to make modiﬁcations to our truncation methods in order toimprove performance.

[17]

Acknowledgments This work was supported by JSPS
KAKENHI Grant Numbers 22H03905 and 21H03775.

[18]
[19]
[20]

References[1] 総務省統計局. 書籍新刊点数と平均価格, 2022.
[2] 文化庁. 平成 30 年度「国語に関する世論調査」の結果について,
2019.
[3] Fusheng Wei, Han Qin, Shi Ye, and Haozhen Zhao. Empirical study of deep learning for text classification in legal
document review. In 2018 IEEE International Conference
on Big Data (Big Data), pages 3317–3320. IEEE, 2018.
[4] David Chen, Hans-Michael Müller, and Paul W Sternberg.
Automatic document classification of biological literature.
BMC bioinformatics, 7(1):1–11, 2006.
[5] Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, and Ziqing
Yang. Pre-training with whole word masking for chinese
bert. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3504–3514, 2021.
[6] Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina
Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of
NAACL-HLT, pages 4171–4186, 2019.
[7] Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint
arXiv:2004.05150, 2020.
[8] Nut Limsopatham. Effectively leveraging bert for legal document classification. In Proceedings of the Natural Legal
Language Processing Workshop 2021, pages 210–216, 2021.
[9] Ashutosh Adhikari, Achyudh Ram, Raphael Tang, andJimmy Lin. Docbert: Bert for document classification.
arXiv preprint arXiv:1904.08398, 2019.
[10] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia
Polosukhin. Attention is all you need. Advances in neural
information processing systems, 30, 2017.
[11] Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does bert look at? an analysis ofbert s attention. In Proceedings of the 2019 ACL Workshop

[21]

[22]

[23]
[24]

[25]

[26]

BlackboxNLP: Analyzing and Interpreting Neural Networks
for NLP, pages 276–286, 2019.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey,
Joshua Ainslie, Chris Alberti, Santiago Ontanon, PhilipPham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big
bird: Transformers for longer sequences. Advances in Neural Information Processing Systems, 33:17283–17297, 2020.
Jun Kong, Jin Wang, and Xuejie Zhang. Hierarchical bertwith an adaptive fine-tuning strategy for document classification. Knowledge-Based Systems, 238:107872, 2022.
Snehal Ishwar Khandve, Vedangi Kishor Wagh, Apurva Dinesh Wani, Isha Mandar Joshi, and Raviraj Bhuminand
Joshi. Hierarchical neural network approaches for long document classification. In 2022 14th International Conference
on Machine Learning and Computing (ICMLC), pages 115–
119, 2022.
Jun He, Liqun Wang, Liu Liu, Jiao Feng, and Hao Wu.
Long document classification from local word glimpses via
recurrent attention learning. IEEE Access, 7:40707–40718,
2019.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler.
Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer
vision, pages 19–27, 2015.
Shanshan Yu, Jindian Su, and Da Luo. Improving bertbased text classification with auxiliary sentence and domain
knowledge. IEEE Access, 7:176600–176612, 2019.
ほりえる. デウス・エクス・マキナを殺せ, 2020.
田山照巳. 伝説の勇者様！
？ ……いえ、補助職の僧侶です, 2020.
Spyridon Samothrakis and Maria Fasli. Emotional sentence annotation helps predict fiction genre. PloS one,10(11):e0141922, 2015.
Jade Goldstein, Mark Kantrowitz, Vibhu Mittal, and Jaime
Carbonell. Summarizing text documents: Sentence selection and evaluation metrics. In Proceedings of the 22nd
annual international ACM SIGIR conference on Researchand development in information retrieval, pages 121–128,
1999.
Jahna Otterbacher. Gender, writing and ranking in review
forums: a case study of the imdb. Knowledge and information systems, 35(3):645–664, 2013.
電撃文庫. 第 30 回電撃大賞 電撃小説大賞 応募要項, 2023.
Yoon Kim. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP),
pages 1746–1751, Doha, Qatar, October 2014. Association
for Computational Linguistics.
Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex
Smola, and Eduard Hovy. Hierarchical attention networks
for document classification. In Proceedings of the 2016 conference of the North American chapter of the association
for computational linguistics: human language technologies, pages 1480–1489, 2016.
Ashutosh Adhikari, Achyudh Ram, Raphael Tang, andJimmy Lin. Rethinking complex neural network architectures for document classification. In Proceedings of the 2019
Conference of the North American Chapter of the Association for Computational Linguistics: Human Language
Technologies, Volume 1 (Long and Short Papers), pages
4046–4051, 2019.