Question: How many words of Google News do the authors use in the training?
Evidence: Deep neural models recently have achieved remarkable results in computer vision BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , and a range of NLP tasks such as sentiment classification BIBREF4 , BIBREF5 , BIBREF6 , and question-answering BIBREF7 . Convolutional neural networks (CNNs) and recurrent neural networks (RNNs) especially Long Short-term Memory Network (LSTM), are used wildly in natural language processing tasks. With increasing datas, these two methods can reach considerable performance by requiring only limited domain knowledge and easy to be finetuned to specific applications at the same time.
CNNs, which have the ability of capturing local correlations of spatial or temporal structures, have achieved excellent performance in computer vision and NLP tasks. And recently the emerge of some new techniques, such as Inception module BIBREF8 , Batchnorm BIBREF9 and Residual Network BIBREF3 have also made the performance even better. For sentence modeling, CNNs perform excellently in extracting n-gram features at different positions of a sentence through convolutional filters.
RNNs, with the ability of handling sequences of any length and capturing long-term dependencies, , have also achieved remarkable results in sentence or document modeling tasks. LSTMs BIBREF10 were designed for better remembering and memory accesses, which can also avoid the problem of gradient exploding or vanishing in the standard RNN. Be capable of incorporating context on both sides of every position in the input sequence, BLSTMs introduced in BIBREF11 , BIBREF12 have reported to achieve great performance in Handwriting Recognition BIBREF13 , and Machine Translation BIBREF14 tasks.
Generative adversarial networks (GANs) BIBREF15 are a class of generative models for learning how to produce images. Basically, GANs consist of a generator G and a discriminator D, which are trained based on game theory. G maps a input noise vector to an output image, while D takes in an image then outputs a prediction whether the input image is a sample generated by G. Recently, applications of GANs have shown that they can generate promising results BIBREF16 , BIBREF17 . Several recent papers have also extended GANs to the semi-supervised context BIBREF18 , BIBREF19 by simply increasing the dimension of the classifier output from INLINEFORM0 to INLINEFORM1 , which the samples of the extra class are generated by G.
In this paper, We proposed an end-to-end architecture named AC-BLSTM by combining the ACNN with the BLSTM for sentences and documents modeling. In order to make the model deeper, instead of using the normal convolution, we apply the technique proposed in BIBREF8 which employs a INLINEFORM0 convolution followed by a INLINEFORM1 convolution by spatial factorizing the INLINEFORM2 convolution. And we use the pretrained word2vec vectors BIBREF20 as the ACNN input, which were trained on 100 billion words of Google News to learn the higher-level representations of n-grams. The outputs of the ACNN are organized as the sequence window feature to feed into the multi-layer BLSTM. So our model does not rely on any other extra domain specific knowledge and complex preprocess, e.g. word segmentation, part of speech tagging and so on. We evaluate AC-BLSTM on sentence-level and document-level tasks including sentiment analysis, question type classification, and subjectivity classification. Experimental results demonstrate the effectiveness of our approach compared with other state-of-the-art methods. Further more, inspired by the ideas of extending GANs to the semi-supervised learning context by BIBREF18 , BIBREF19 , we propose a semi-supervised learning framework for text classification which further improve the performance of AC-BLSTM.
The rest of the paper is organized as follows. Section 2 presents a brief review of related work. Section 3 discusses the architecture of our AC-BLSTM and our semi-supervised framework. Section 4 presents the experiments result with comparison analysis. Section 5 concludes the paper.
And we use the pretrained word2vec vectors as the ACNN input, which were trained on 100 billion words of Google News to learn the higher-level representations of n-grams.
Answer: 100 billion words.