Title: Compact Encoding of Words for Efficient Character-level Convolutional Neural Networks Text Classification
Authors: Wemerson Marinho
Luis Marti
Nayat Sanchez-pi
Venue: ICLR 2018
Keywords: Character Level Convolutional Networks
Text Classification
Word Compressing

Abstract: This paper puts forward a new text to tensor representation that relies on information compression techniques to assign shorter codes to the most frequently used characters. This representation is language-independent with no need of pretraining and produces an encoding with no information loss. It provides an adequate description of the morphology of text, as it is able to represent prefixes, declensions, and inflections with similar vectors and are able to represent even unseen words on the training dataset. Similarly, as it is compact yet sparse, is ideal for speed up training times using tensor processing libraries. As part of this paper, we show that this technique is especially effective when coupled with convolutional neural networks (CNNs) for text classification at character-level. We apply two variants of CNN coupled with it. Experimental results show that it drastically reduces the number of parameters to be optimized, resulting in competitive classification accuracy values in only a fraction of the time spent by one-hot encoding representations, thus enabling training in commodity hardware.

TL;DR: Using Compressing tecniques to Encoding of Words is a possibility for faster training of CNN and dimensionality reduction of representation

Review discussion:
The paper proposed to encode text into a binary matrix by using a compressing code for each word in each matrix row. The idea is interesting, and overall introduction is clear.
However, the work lacks justification for this particular way of encoding, and no comparison for any other encoding mechanism is provided except for the one-hot encoding used in Zhang & LeCun 2015. The results using this particular encoding are not better than any previous work.
The network architecture seems to be arbitrary and unusual. It was designed with 4 convolutional layers stacked together for the first layer, while a common choice is to just make it one convolutional layer with 4 times the output channels. The depth of the network is only 5, even with many layers listed in table 5.
It uses 1-D convolution across the word dimension (inferred from the feature size in table 5), which means the convolutional layers learn intra-word features for the entire text but not any character-level features. This does not seem to be reasonable.
Overall, the lack of comparisons and the questionable choices for the networks render this work lacking significance to be published in ICLR 2018.
Thank you for your comments.
We created an updated version where we did our best to improve the quality of our presentation and express it in a clear way.
In this new version, we provide a better description and justification of our proposal.
In particular, we discuss why we took some decisions regarding the encoding.
Similarly, we now include an additional neural network architecture in the comparative experiments. For completeness, we included a more detailed description of datasets and base models involved in the experiments.
Regarding encodings comparison, in the initial stage of our research, we investigated others encodings, mainly Huffman and End Tag Dense Codes- ETDC. We discarded these because we did not found a way to represent words in a distinct way once we concatenate each char code. We made more clear this step in the manuscript and still working in a way to modify ETDC to make it distinct for each word.
Our main concern with this paper is show that a more compact encoding procedure could reduce the computational footprint of using words codified character-by-character in text classification.  Character-based text classification allows to handle less curated texts or texts in languages that have a lot of declensions and with poor or none a priori pre-trained language modules (ala word2vec).
In terms of classification performance, we have matched the results of Zhang & LeCun (2015) which represent the state of the art in this area and the departing point of this work.
It should be noted that like them, we also beat traditional methods. In this regard, in order to improve the readability of the paper, we changed the comparison metric to accuracy, different of Zhang & LeCun (2015) that report error loss ($(1-accuracy)\times100$).
On the other hand, in terms of computational footprint, our approach is much faster than Zhang & LeCun (2015). We find that is is a relevant result as it makes the approach suitable for extended use. We are providing along with our paper the supporting code. We expect that with the collaboration of the community a streamlined implementation can be obtained and even better times will be reached.
To our knowledge, our approach is the first who try to codify words into vectors using only characters composition in a sparse way. We are aware that there is room for improvement. In the process of updating the paper, we included another neural network with positive results.
Nevertheless, this direction should be properly explored in the future. In this paper, we focused mainly on the comparison with the previous results by Zhang & Lecun (2015) but we are confident that other architectures will yield better results.
Our main line of research is educational data mining where it is crucial to be able to handle texts produced by students, with equations, orthographic errors and not-so-formal language. In this scenario, we have a lot of interest in building better and faster solutions build upon the character-based approach originally put forward by Zhang & Lecun (2015).
We appreciate that you take a moment an revise again that paper under the light of these comments and modifications.
The manuscript proposed to use prefix codes to compress the input to a neural network for text classification. It builds upon the work by Zhang & LeCun (2015) where the same tasks are used.
There are several issues with the paper and I cannot recommend acceptance of the paper in the current state.
- It looks like it is not finished.
- the datasets are not described properly.
- It is not clear to me where the baseline results come from.
They do not match up to the Zhang paper (I have tried to find the matching accuracies there).
- It is not clear to me what the baselines actually are or how I can found more info on those.
- the results are not remarkable.
Because of this, the paper needs to be updated and cleaned up before it can be properly reviewed.
On top of this, I do not enjoy the style the paper is written in, the language is convoluted.
For example: “The effort to use Neural Convolution Networks for text classification tasks is justified by the possibility of appropriating tools from the recent developments of techniques, libraries and hardware used especially in the image classification “
I do not know which message the paper tries to get across here.
As a reviewer my impression (which is subjective) is that the authors used difficult language to make the manuscript look more impressive.
The acknowledgements should not be included here either.
Thank you very much for your time and constructive comments. We have addressed the issues pointed out in your remarks and tried to make more evident the contributions of our work. We made important improvements on the quality of the text, the description of our proposal and presentation of the experimental results.
In this new version, we provide a better description and justification of our proposal.
In particular, we discuss why we took some decisions regarding the encoding.
Similarly, we now include an additional neural network architecture in the comparative experiments. For completeness, we included a more detailed description of datasets and base models involved in the experiments.
Our main concern with this paper is show that a more compact encoding procedure could reduce the computational footprint of using words codified character-by-character in text classification.  Character-based text classification allows to handle less curated texts or texts in languages that have a lot of declensions, and with poor or none a priori pre-trained language modules (ala word2vec).
In terms of classification performance, we have matched the results of Zhang & LeCun (2015) which represent the state of the art in this area and the departing point of this work.
It should be noted that like them, we also match traditional methods. In this regard, in order to improve the readability of the paper we changed the comparison metric to accuracy, different of Zhang & LeCun (2015) that report error loss ($(1-accuracy)\times100$).
On the other hand, in terms of computational footprint, our approach is much faster than Zhang & LeCun (2015). We find that is is a relevant result as it makes the approach suitable for extended use. We are providing along with our paper the supporting code. We expect that with the collaboration of the community a streamlined implementation can be obtained and even better times will be reached.
To our knowledge, our approach is the first who try to codify words into vectors using only characters composition in a sparse way. We are aware that there is room for improvement. In the process of updating the paper we included another neural network with positive results.
Nevertheless, this direction should be properly explored in the future. In this paper we focused mainly on the comparison with the previous results by Zhang & Lecun (2015) but we are confident that other architectures will yield better results.
Our main line of research is educational data mining where it is crucial to be able to handle texts produced by students, with equations, orthographic errors and not-so-formal language. In this scenario, we have a lot of interest in building better and faster solutions build upon the character-based approach originally put forward by Zhang & Lecun (2015).
We appreciate that you take a moment an revise again that paper under the light of these comments and modifications.
Regarding the writing style, we really apologize. Sometimes is difficult to express yourself in a foreign language. We did your best in this updated version to not give you these impressions.
After looking at the revision, the manuscript looks in a much better shape at this point.
However, due to the amount of  changes,
I believe it has to go trough a full review process again as I mentioned in the original review.
Therefore I stand by my original opinion the paper cannot be accepted now.
If the authors are thinking to re-submit this manuscript, I think they should focus on the following:
- Check the literature on the datasets and compare to more recent approaches than Zhang & LeCun such that the baselines are the current state of the art. I am not so familiar with these datasets so I do not know the current best approaches for these datasets.
- Polish the language. After quickly reading through the manuscript, I found several more strange formulations.
- The times per experiment from Zhang & LeCun should be replaced by current (efficient) re-implementations on the same hardware. Since 2015, we have made advances in hardware and software libraries.
Therefore the measurements from 2015 and 2017/2018 are not directly comparable.
This paper proposes a new character encoding scheme for use with character-convolutional language models. This is a poor quality paper, is unclear in the results (what metric is even reported in Table 6), and has little significance (though this may highlight the opportunity to revisit the encoding scheme for characters).
Thank you for your time for helping us to better express our findings.
We made important improvements in the quality of the text, the description of our proposal and presentation of the experimental results.
In this new version, we provide a better description and justification of our proposal.
In particular, we discuss why we took some decisions regarding the encoding.
Similarly, we now include an additional neural network architecture in the comparative experiments. For completeness, we included a more detailed description of datasets and base models involved in the experiments.
Our main concern with this paper is show that a more compact encoding procedure could reduce the computational footprint of using words codified character-by-character in text classification.  Character-based text classification allows to handle less curated texts or texts in languages that have a lot of declensions and with poor or none a priori pre-trained language modules (ala word2vec).
In terms of classification performance, we have matched the results of Zhang & LeCun (2015) which represent the state of the art in this area and the departing point of this work.
It should be noted that like them, we also beat traditional methods. In this regard, in order to improve the readability of the paper, we changed the comparison metric to accuracy, different of Zhang & LeCun (2015) that report error loss ($(1-accuracy)\times100$).
On the other hand, in terms of computational footprint, our approach is much faster than Zhang & LeCun (2015). We find that is is a relevant result as it makes the approach suitable for extended use. We are providing along with our paper the supporting code. We expect that with the collaboration of the community a streamlined implementation can be obtained and even better times will be reached.
To our knowledge, our approach is the first who try to codify words into vectors using only characters composition in a sparse way. We are aware that there is room for improvement. In the process of updating the paper, we included another neural network with positive results.
Nevertheless, this direction should be properly explored in the future. In this paper, we focused mainly on the comparison with the previous results by Zhang & Lecun (2015) but we are confident that other architectures will yield better results.
Our main line of research is educational data mining where it is crucial to be able to handle texts produced by students, with equations, orthographic errors and not-so-formal language. In this scenario, we have a lot of interest in building better and faster solutions build upon the character-based approach originally put forward by Zhang & Lecun (2015).
We appreciate that you take a moment an revise again that paper under the light of these comments and modifications.

Meta-review:
role: PC
data: comment: meta score: 4

The paper has been extensively edited during the review process - the edits are so extensive that I think the paper requires a re-review, which is not possible for ICLR 2018

Pros:
 - potentially interesting and novel approach to prefix encoding for character level CNN text classification
 - some experimental comparisons
Cons:
 - lacks good comparison with the state-of-the-art, which makes it difficult to determine conclusions
 - writing style lacks clarity.

I would recommend that the authors continue to improve the paper and submit it to a later conference.


Decision: Reject