Learning to Control Latent Representations for Few-Shot Learning of Named Entities
Authors: Omar U. Florez
Erik Mueller
Venue:  (2020)
Keywords: Memory management
neuroscience
reinforcement learning
learning with small data

Abstract: Humans excel in continuously learning with small data without forgetting how to solve old problems.
However, neural networks require large datasets to compute  latent representations across different tasks while minimizing a loss function. For example, a natural language understanding (NLU) system will often deal with emerging entities during its deployment as interactions with users in realistic scenarios will generate new and infrequent names, events, and locations. Here, we address this scenario by introducing a RL trainable controller that disentangles the  representation learning  of a  neural encoder from its memory management role. 

Our proposed solution is straightforward and simple: we train a controller to execute an optimal sequence of read and write operations on an external memory with the goal of  leveraging  diverse activations from the past and provide accurate predictions. Our approach is named Learning to Control (LTC) and allows few-shot learning with two degrees of memory plasticity.  We experimentally show that our system obtains accurate results for few-shot learning of entity recognition in the Stanford Task-Oriented Dialogue dataset.

Under review as a conference paper at ICLR 2020
LEARNING TO CONTROL LATENT REPRESENTATIONS FOR FEW-SHOT LEARNING OF NAMED ENTITIES
Anonymous authors Paper under double-bling review
ABSTRACT
Humans excel in continuously learning with small data without forgetting how solve old problems. However, neural networks require large datasets to compute latent representations across different tasks while minimizing loss function. For example, a natural language understanding (NLU) system will often deal with emerging entities during its deployment interactions with users in realistic scenarios will generate new and infrequent names, events, and locations. Here, we address this scenario by introducing RL trainable controller that disentangles the representation learning of neural encoder from its memory management role.
Our proposed solution is straightffrward and simple: train controller to execute an optimal sequence of read and write operations on an external memory with the goal of leveraging diverse activations from the past and provide accurate predictions. Our approach is named Learning to Control (LTC) and allows few-shot learning with two degrees of memory plasticity. We experimentally show that system obtains accurate results for few-shot learning of entity recognition in the Stanford Task-Oriented Dialogue dataset.
MOTIVATION
Today, supervised models have problems incorporating new tasks over time while protecting previ- ously acquired knowledge. This is because these algorithms require that all data given prior to training. This becomes a problem in the presence more general scenarios in which new classes emerge during training data exhibit long-tailed distributions. Hence, classes with large sup- port dominate the learning of gradient-based representations causing the catastrophic forgetting of under-representted classes, Kirkpatrick et al. 2016).
Unlike deep neural networks, humans and other mammals exel in learn incrementally with small data. Biological evidence suggests that the process of acquiring new skills occurs in different brain areas with at least two degrees of plasticity, Wixted et al. (2018): : 1) A slow memory that is dense and requires extensive practive. A fast memory that sparse and stores volatile information. Inspired by these observations, we introduce a trainable controller, Learning to Control (LTC), that learns to interact with both a slow memory (consisting neural encoders) and a fast memory (consisting of an associative array storing key-value pairs). We experimentally show the advantage of using LTC for few-shot learning setup. Figure depicts the network architecture our model.
We present the following contributions:
We propose a novel architecture that uses a trainale controller to manipulate latent represen- tations in an external memory (Section 3
We introduce the use of reward signal that is proportional to the average reduction of entropy when attending the memory entries. This enables us to propose reinforcement learning approach that learss policy based on interactions with the external memory.
We show the generality our solution for the few-shot learning of entities in the Stanford Task-Oriented Dialogue dataset.

---

Under review as a conference paper at ICLR 2020
2 DENSE AND SPARSE MEMORIES
The proposed architecture considers the use two types of memories that behave differently during backpropagation. First, we define a memory follows.
Definition 2.1. (Memory) Given neural model far y, () that maps observation to the learning task and is parameterized by 0. Then, the memory of f(x, y, 0) is a collection of co-activations responding similarly to the reocurring patterns associated the learning task y.
A Neural Network uses most of its memory to remember patterns in the dataset. However, our approach distinguishes between dense and sparse memories.
Definition 2.2. (Dense Memory) A dense memory is type of memory, in which all its learnable parameters 0 have the possibility of being updated during training.
We want to preserve latent representations of the input between training steps but a dense memory will likely overwrite them based on frequent global updates. We then need a sparse memory with addressable memory entries and sparse updates.
Definition 2.3. (Sparse Memory) Given a model f(x y, ) let a(x, y) be a memory operation which returns & updating parameters to incorporate the pair (z, v). Then, a sparse memory is type of memory that satisfies the sparse updating constraint:
|a(x, v)l < |0]
where [6] is the total number of trainable parameters.
We use the sparse memory introduced in Kaiser al. 2017) which consists of two arrays of size mem__iz ze for storing keys (K) and values (V). Each key has a dimensionality of key_ dim and each value is scalar representing a class label. In our experiments, we fixed the number of updating parameters, i.e. keys, to be very small (k 10) in comparison to mem_size ze in order to satisfy the sparse updating constraint, < mem_size. The final form of the sparse memory as follows.
M
K mem mem__izze dim, V theem_size
The state of a sparse memory changes based on the following memory operations.
Definition 2.4. (Memory Operations) Let s be a input embedding that we want to incorporate into M, be class label, and be the memory index of the most similar key to based on the Cosine similarity.
1. If Vi A y. the write(i, 8, v) operation registers new class with M[1] (s, v).
2. If V = y. the update(i, 8, v, operation modifies the stored key-based representation K[i] with M[i] = (I) [[i] + s|l v).
LEARNING TO CONTROL
The goal of Learning To Control (LTC) is learn strategy manage the entries of sparse memory. We characterize this decision process with a memory policy Te which represents the probability of running an memory operation given the current state of the memory. Rathern than following heuristics (e.g., overwritting in the oldest memory entry as in Kaiser et al. (217)), learning such policy has the advantage of optimizing the allocation of infrequent classes considering the limited number of memory entries in M. Indeed, To transitions M to a new configuration in which its keys are more adapted store and maintain informative latent vectors. To clarify the connection between the components, let us consider the forward and backward computations involved in LTC.
3.1 FORWARD COMPUTATION
As illustrated in Figure | information flows through the hidden layers the neural encoder, feeds the controller, and reaches the external memory M. The encoder is Long-Short Term Memory (LSTM)
2

---

Under review as a conference paper at ICLR 2020
Policy T
Keys
Values
S
a s, y)
Observation
LSTM
r(s, a)
X
Neural
External
Controller
Encoder
Memory
Figure I: Architecture of our approach based on neural encoder that generates input representations and a trainable controller that learns to store and maintain them in an external memory formed by key-value pairs. The memory is modified via operations each of which returns reward signal
neural network that transforms the raw input into the last hidden state of the following recurrence h; LSTT (z, h4-1). where s h; the state of the environment prior any memory operation and corresponds to the last hidden state of the LSTM encoder.
Which memory entries should the controller update incorporate the learning task (s, y)? Our approach is to induce the probablity distribution No (ils) over the memory entries given the state and sample from there the most likely memory index im
To (i) Softmar(s K2)
i mg ~ (Fo
Then, the controller exccutes the memory operation a(ix. s, y) at position is considering the Definition 2.4 During inference, V[ returns the predicted class of the raw input
3.2 BACKWARD COMPUTATION
The forward step performs an operation 8,3) in the sparse memory and modifies its keys integrate latent representations. The backward step estimates the objective function (()) and computes its gradients with respect to the trainable parameters of the encoder and controller modules.
In the case of the controller, we stochastic policy in a direction that maximizes (@) during few-shot learning. For our purposes, each memory operation a returns reward that measures the efficacy of this operation reduce uncertainty about which of the keys in K will lead the transition between memory states. This means going from a state in which no operation took place in memory (K,-1) an updated state (K,). More formaly, our reward function r, (s, a) corresponds to the reduction of entropy H when softly attending over the memory entries before and after executing the operation a.
compute the gradient vector VoJ(O) that updates the
bg. K₁
att = (ooftmar(hkk)
We the dot-product attention score Luong, Pham, and Manning 2015) compute the logits hg, over the memory keys and use Softmax to generate the probability distribution attk,- ZZu et al. (2005) demonstrates that models that show high entropy in their output distributions often fail to discriminate classes because of exhibiting uniform- like distributions and high degree of uncertainty. Thus, we use the following entropy-basee based reward signal to encourage the use of few memory entries to modify the sparse memory M.
r (H (attk.) Hatt
Based memory operations, approximate the objective function from batch of sampled episodes of length as the expected value of the cumulative sum of rewards under the policy na.
T
((0) = Exa
Σ
rr(s, a)
3

---

Under review as a conference paper at ICLR 2020
Model
5-way 1-shot
5-way 5-shot
20-way 1-shot
20-way 5-shot
Memory Augmented NN
63.1%
66.7%
$7.2%
61.3%
Matching Networks
67.3%
71.4%
62.2%
68.3%
Learning Control
71.5%
77.1%
65.3%
75.8%
Table 1: Average accuracy for few-shot learning in the STDO dataset.
The REINFORCE algorithm Williams 1992) allow us directly take the gradients of the objective function J(O) as described in Equation and used them to perform backpropagation.
VoJ(O)
1
Σ Ve log no (1h)))
Σ
4 EXPERIMENTS
We evaluate our proposed method on the Stanford Task-Oriented Dialogue Dataset (STDO) Eric et al. (017) which consists of 031 dialogues in the domain of an in-car assistant that provides automatic responses to the requests of driver considering weather, point-of-interrst, and scheduled domains.
4.1 BASELINES
We compare Learning to Control (LTC) with the following baselines: 1) Matching Networks (MN), Vinyals al. (2016): This algorithm provides one-shot learning capabilities by jointly mapping a small labeled support set and an unlabeled example to its most likely label, and 2) Memory Augmented Neural Networks (MANN), Santoro et al. (2016): This algorithm is designed to provide one-shot learning based on Neural Turing Machine and a curriculum training regime.
4.2 FEW-SHOT LEARNING FOR ENTITY RECOGNITION
We study the problem of sequence labeling find the best label sequence (named entities) for a given input sentence considering an LSTM encoder augmented with an external memory to store hidden representations for each recurrent unit. We use the Stanford Named Entity Recognizer (NER 1 to augment the STDO dataset with 7 classes (Location, Person, Organization, Money, Percent, Date, Time) and an non-entity class. This results in 15, 928 observations, 8 classes, and an average sequence length of 44 words.
We change the format of the STOD dataset to simulate a scenario in which classes obtain incremental support during training. This is done by forming episodes of labeled examples that show uniform distribution over classes (e.g.. 5-way), so we can track the number of times a particular class was presented to the model during training (e.g., 1-shot learning). This format was proposed by (Santoro et al. (2016) to study few-shot learning. The goal of this experiment is to learn with small data, so study the learning behavior of L.TC with an external memory of 1, 000 entries and in relation to two models designed to provide few-shot learning capabilities: Memory Augmented Neural Networks, Santoro al. (016) and Matching Networks, Vinyals al. (2016).
Table 1 shows that LTC provides an average improvement of +5.2 in the STOD dataset with respect the MN neural model, which is consistently the second most accurate option for both 5-way and 20-way classification and when the number of classes ranges from small (5-way) to large (20-way). Similarly, LTC also shows an average improvement + +0.35% with respect the MANN model. For the scenario of learning with infrequent classes, LTC outperforms MN when only training observation presented (I-shot) and and 20 classes are available during training by +4.2% and +3.1%, respectively. The performance advantage in the results be explained by the extra memory capacity of LTC and represent an opportunity to explore later what is the minimum memory size to continuously learn with small data, an more challenging problem to study in the future.
https://hllp.stanforrd so hwwp:///eddddddotoaeeeeeeeeeeee
html
4

---

Under review as a conference paper at ICLR 2020
5 RELATED WORK
The idea of training a controller that interacts with a memory to incrementally allocate distributional representations builds upon a wide range of research in machine learning and memory augmented algorithms.
Kraska et al. (2018) introduced the concept of learned indices structures as a learning architecture that predicts the position or existence of records. Their main goal is to prevent too many collisions from being mapped to the same position inside the hash index. This is because collisions often cause memory overhead when traversing a linked list or require the allocation of additional memory for storing more records. They approach this problem as supervised learning of the cumulative distribution function of hash keys, which leads to minimizing the number of collisions. In contrast, we propose the use of an associative array as an indexed structure that supports generalization, so the collision of multiple observations to a similar hash key if adequate for predicting the class of similar input data. While the idea of learning hash functions as neural networks is not new, existing work mainly focused on learning a better hash-fur mash-function to map observations into low-dimensional embeddings for similarity search assuming a fixed data distribution Qian et al. (2014). To our knowledge, it has not been explored yet if it is possible to learn hash index according to data distribution in which few training observations are presented per class during training.
Deep Neural Networks are models that solve classification problems with non-linearly separable classes. These are shown to be good for problems such as object detection Krizhevsky, Sutskever and Hinton (017)). language translation Johnson al. (016) and image generation Goodfellow et al. (2014). Recently, recurrent neural networks (RNN) have been studied to manage the states between time steps using attention mechanism that addresses similar content [Bahdanau, Cho. and Bengio (2014). Some similar approaches have also studied the problem of few-shot learning |Santoro al. (2016]Koch, Zemel, and Salakhutdinov (2015 Vinyals et al. (2016), a representative example is Matching Networks Vinyals et (2016), which uses an attention mechanism to augment neural networks for set-to-set learning.
A relevant research topic to this work is the idea of memory augmented networks that extend its capabilities by coupling an external memory. For example, Kaiser et al. (2017) also proposes key-value memory, but with no controller mechanism for adequate training of few-shot learning tasks. Also, Neural Turing Machines ANTM)Gravee Wayne, and Danihelka (2014) are differentiable architectures allowing efficient training with gradient descend and showing important properties for associative recall for learning different sequential patterns. Although they have important properties for one-shot learning given their sequential memory management, the supervised nature of its training still shows issues related to catastrophic forgetting.
6 CONCLUSIONS
Learning new tasks without forgetting the previously ones references the plasticity of our own brain to retain previously acquired knowledge. In this work, we show learning system that stores information about named entities with two levels of representation: dense and sparse. While a dense memory captures features that are shared across tasks, sparse memory incorporates embedding vectors and prevents them from being overwritten by global updates during backpropagation. To do this, we rely on the sparse structure of associative arrays to locally update a small set of memory entries and on the use of trainable controller that manages the transition of the sparse memory to a state of low entropy that also addresses the learning task. This enables a model to incrementally learn in the presence of a few observations per class. We experimentally show that our system obtains accurate results for few-shot learning of entities in the Stanford Task-Oriented Dialogue dataset in comparison with state-of-the-an few shot learning algorithms (Memory Augmented Neural Networks and Matching Networks).
REFERENCES
Bahdanau, D.; Cho, K.; and Bengio, Y. 2014. Neural machine translation by jointly learning to align and translate. CoRR abs/1409.0473.
5

---

Under review as a conference paper at ICLR 2020
Eric, M.; Krishnan, L.; Charette, F.; and Manning, C. 2017. Key-value retrieval networks for task-oriented dialogue. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, 37-49. Saarbrücken, Germany: Association for Computational Linguistics.
Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; and Bengio. Y. 2014. Generative adversarial nets. In Advances in Neural Information Processing
Systems 27. 2672-2680.
Graves, A.; Wayne, G.; and Danihelka, 2014. Neural turing machines. CoRR abs//4al..50 1410.5401
Johnson, M.; Schuster, M.; Lc, Q. V.: Krikun, M.; Wu, Y; Chen, Z.; Thorat, N.; Viégas, F:; Wattenberg, M.; Corrado, G.; Hughes, M.; and Dean, J. 2016. Google's multilingual neural machine translation system: Enabling zero-shot translation. Technical report, Google.
Kaiser, L.; Nachum, O.; Roy, A.; and Bengio, S. 2017. Learning to remember rare events,
Kirkpatrick, J.; Pascanu, R.; Rabinowitz, N. C.; Veness, J.; Desjardins, G.; Rusu, A. A.; Milan, K.; Quan, J.: Ramalho, T.: aarbnkkkaa A.; Hassabis, D.; Clopath, C.; Kumaran, D.; and Hadsell, R. 2016. Overcoming catastrophic forgetting in neural networks. CoRR.
Koch, G.; Zemel, R.; and Salakhutdinov, R. 2015. Siamese neural networks for one-shot image recognition. PhD Thesis.
Kraska, T.; Beutel, A.; Chi, E. H.; Dean, J.; and Polyzotis, N. 2018. The case for learned index structures.
Krizhevsky, A.; Sutskever, I.; and Hinton, G. 2017. Imagenet classification with deep convolutional neural networks. Commun. ACM 60(6).
Luong, T.; Pham, H.; and Manning, C. D. 2015. Effective approaches to attention prrcntion-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, 1422-1421. Lisbon, Portugal: Association for Computational Linguisties.
Qian, Q:: Hu, J.: Jin, R.; Pei, and Zhu, S. 2014. Distance metric learning using dropout: A structured regularization approach. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 14, 323-332.
Santoro, A.; Bartunov, S.; Botvinick, M.; Wierstra, D.; and Lillicrap, T. 2016. Meta-learning with memory-augmented neural networks. In Proceedings of the 33rd International Conference International Conference on Machine Learning Volume 48, ICML' 16, 1842-1850.
Vinyals, O.: Blundell, C.; Lillicrap, T.; kavukcuoglu, k.; and Wierstra, D. 2016. Matching networks for one shot learning. In Advances in Neural Information Processing Systems.
Williams, R. J. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement comneciooniserriffoeeeeen learning. Mach. Learn. 8(3-4).
Wixted, J. T.; Goldinger, S. D.; Squire, L. R.; Kuhn, J. R.; Papesh, M. H.; Smith, K. A.; Treiman, D. M.; and Steinmetz, P. N. 2018. Coding of episodic memory in the human hippocampus. Proceedings of the National Academy of Sciences.
Zhu, S.; Ji, X.; Xu, W.; and Gong, Y. 2005. Multi-labelled classification using maximum entropy method. In Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval.


Reviews:
[{"review_id": "rJlyw3yTKr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors present a method, Learning to Control (LTC), that enables a reinforcement learning agent to learn to read and write external memory. They follow the intuition that human has two degrees of plasticity for memory, which leads to the dense-sparse memory design in this paper. The proposed method can be applied to a few-shot setting. \n\nwriting: The writing is ok. Readers can obtain the essence by a reasonable amount of guessing. \n\n\nStrength: Although memory augmented neural networks and reinforcement learning are both well-known, the proposed method combined them in a novel way.  The presented approach outperforms two baselines on the stanford dialogue dataset. \n\nWeakness: There is no ablation study for the proposed method to fully understand why the proposed method is superior. The authors use REINFORCE as the RL algorithm which is no longer useful in most of the complex RL tasks. More advanced RL algorithms are encouraged to be tried. The proposed title is \"Learning to Control\",   which I don't see where the control part is. It seems the LTC only refers to the REINFORCE agent. The experiments are very minimal which did not provide enough information to distinguish the proposed method from other methods. \n\nOverall, the general direction is promising. However, the paper is not yet fully finished. I believe the authors need major changes to the experiments section as well as the method description section. Hence, I cannot recommend this paper to acceptance.\n\nminor: \nexel -> excel \nFigure 1 needs more clarification.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "r1gvaG_X9B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors build on work regarding few-shot learning with memory augmented networks, specifically [Kaiser, et al., ICLR17] where the goal is to learn a memory address mapping such that generalization is achieved by finding the nearest neighbor memory address when predicting the label. For correct predictions, the memory key is updated to include the associated (predicted) address while new memory locations are written for mistakes. Whereas [Kaiser, et al., ICLR17] follows a LRU-like procedure for replacing memory, the current work proposes performing policy-gradient RL where the action space is the memory locations and the reward is reduction of entropy over the memory address assignment distribution over the memory locations. This approach is empirically studied for an RNN approach to NER, specifically considering few-shot learning for NER in the Stanford Task-Oriented Dialogue (STOD) dataset — showing non-negligible improvements over Memory Augmented Networks [Santoro, et al., ICML16] and Matching Networks [Vinyals, et al., NeurIPS16].\n\nFrom a high-level perspective, memory networks have played an important role in building dialogue managers (and in several applications) and their proposed expansion over [Kaiser, et al., ICLR17] in the context of few-shot, sparse memory architectures seems sensible. Additionally, the empirical results o NER seem promising. However, in my opinion, this paper is making contributions along two dimensions, but neither of them convincingly as detailed below. \n\nFirst, one potential point of emphasis is the ‘learning to control’ mechanism through PG RL being better than [Kaiser, et al., ICLR17] using with a LRU-like policy. To do this, it would make sense to both compare to the datasets in the previous work (Omniglot, WMT — even if it does require additional architectures that are pretty straightforward expansions) and to include this architecture in the submitted paper as there is a publicly available dataset (https://github.com/himani-arora/learning_to_remember_rare_events). While the synthetic dataset probably isn’t necessary, one could easily think of synthetic datasets that would clearly contrast with the existing method and conduct ablation/configuration studies regarding different number of neighbors, size of memory, exact vs. approximate nearest neighbor for lookup, etc. \n\nA second point of interest is the NER task. From this perspective, there is some existing work (e.g., [Fritzler, Logacheva & Kretov, SAC19; Hou, et al., https://arxiv.org/pdf/1906.08711.pdf (Not required as not accepted yet); Hofer, et al., https://arxiv.org/pdf/1811.05468.pdf (Not clear if accepted)]. While these are largely preliminary works, they do provide datasets (and there are many others) for NER that would potentially make a convincing case for this being a state-of-the-art approach for few-shot learning in NER. However, I am not sure that being the state-of-the-art for few-shot NER would be a sufficient contribution for ICLR, even if done convincingly. If there was something in the algorithm specific to sequence prediction, then I think few-shot sequence prediction would be a sufficient contribution, but given the current architecture, I would think that the bar is a methodological contribution to adding sparse memory units (via RL in this case) is the acceptable level of contribution.\n\nFinally, my belief that the contribution is too narrow and not sufficiently developed is supported by the writing seeming rushed. The paper makes much more sense after reading [Kaiser, et al., ICLR17] as Sections 2, 3 of the submission are missing the aspects of a clear constrast to [Kaiser, et al., ICLR17], a working example to provide readers a more intuitive understanding, a clear example showing dimensionality and notation (beyond what is in Figure 1). There are also many typos (e.g., “mammals exel”, “trainale controller”, “Rathern then”, amongst others) and general need for proofreading. However, this is not the determining factor in the paper. Basically, I think the contribution is insufficient in terms of scope and convincingness of the contribution. Thus, I recommend rejecting in its current form — even if the underlying idea is promising and worth developing further.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BJeyg1lTFH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe paper proposes a memory network architecture with a sparse memory. Each memory entry contains a key (vector) and a value (class label). The memory is addressed using a policy pi_theta, which selects a single memory entry to be updated at each time step. The policy pi_theta is trained using policy gradient with the reward being the increase in the policy's certainty (measured as entopy). The model is evaluated on an online NER task that mimics the meta-learning setting of http://proceedings.mlr.press/v48/santoro16.html. In that work, the examples are provided in episodes. The labels are renamed at the beginning of each episode (to prevent fitting to the labels). The model has to predict the label of each example in sequence, and the correct label is given after each prediction.\n\nThe paper suffers from the following issues:\n\n- The method in the paper is very similar to the existing discrete addressing scheme (https://www.mitpressjournals.org/doi/full/10.1162/neco_a_01060), which naturally also uses RL for training. \n\n- The method description is either incomplete or incorrect. The task label does not seem to influence the loss function at training time. The current training loss only encourages the entropy to reduce, so unless the LSTM is pre-trained, there is no guarantee that the key s will address the right memory entry. Please correct me if I am mistaken here.\n\n- The writing is unclear. Task descriptions and notations are not properly set up. The experiment section does not explain the details of the experiments, leaving the reader to look at Santoro et al., 2016 for context. Even then, some statements in the experiment section are confusing. For instance, the numbers of classes in the description (5 and 20) do not match the number of NER classes (8).\n\n- The task used in the paper is non-standard. The paper could have used meta-learning tasks from previous papers, including the ones in Santoro et al., 2016.\n\nOther comments:\n\n- The citations should be written as \"xxx (author, year)\" and not \"xxx, author (year)\".\n- The paper could benefit from proofreading.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes to use policy-gradient RL to learn to read and write actions over memory locations using as reward the entropy reduction of memory location distribution. The authors perform experiments on NER in Stanford Dialogue task, that are framed though as few-shot learning. The reviewers have pointed out shortcomings of the paper with regards to its novelty, narrow contribution in combination thin experimental setup (the authors only look into one dataset and one task with minimal comparison to previous work and no ablation studies as to understand the behaviour of the model) and clarity (method description seems to be lacking some crucial components of the model). As such, I cannot recommend acceptance but I hope the authors will use the reviewers comments to transform this into a strong submission for a later conference.
Decision: Reject