dataesr/scientific-paragraphs-categorization huggingface dataset:
Magnetic Memory MRAM (magnetic RAM) is a memory that uses the magnetism of electron spin to provide non-volatility (Figure  4 ). MRAM stores information in magnetic material integrated with silicon circuitry. Toggle MRAM uses a one transistor, one MTJ (magnetic tunnel junction) cell to provide a simple high-density memory. During a read, the pass transistor is activated and data are read by comparing the resistance of the cell to a reference device. During writes, the magnetic field from Write Line 1 and Write Line 2 writes the cell at the intersection of the two lines but does not disturb other cells on either line. Another MRAM technology uses a spin torque transfer property, which is the manipulation of the spin of electrons with a polarizing current, to establish the desired magnetic state of the free layer to program, or write, the bits in the memory array.
Spin transfer torque MRAM (STT-MRAM) provides a significant reduction in switching energy compared to toggle MRAM and is highly scalable, enabling higher density memory products. The advantage of MRAM is fast switching speed compared to other nonvolatile memories, with ~1-10 ns read and write erase times and very good endurance (up to 10 15 cycles). One major concern with MRAM is the scalability that is difficult due to the complexity of etching many layers with good conformity. Moreover, MRAM etching generally uses ion beam etching, which is not suitable for extensive scaling. The other generally reported issue is MRAM data retention. However, recent material and stack development allowed significantly improved retention performances  [83] , which can be attributed to the adoption of a dual MgO interface instead of the standard single layer.
Spin transfer torque MRAM (STT-MRAM) provides a significant reduction in switching energy compared to toggle MRAM and is highly scalable, enabling higher density memory products. The advantage of MRAM is fast switching speed compared to other nonvolatile memories, with ~1-10 ns read and write erase times and very good endurance (up to 10 15 cycles). One major concern with MRAM is the scalability that is difficult due to the complexity of etching many layers with good conformity. Moreover, MRAM etching generally uses ion beam etching, which is not suitable for extensive scaling. The other generally reported issue is MRAM data retention. However, recent material and stack development MRAM also suffers from small ON/OFF current ratio compared to the other emerging memories: STT-MRAM has a small window and TMR reduces with temperature  [83, 84].
Inserting the emerging memory on the main memory bus, alongside DRAM components, will present significant system benefits. It will improve data integrity management; data can be quickly secured locally, on a word-by-word basis, instead of using current cumbersome journaling or check-pointing schemes to protect data from events such as a loss of power supply. This can significantly reduce data traffic in a data center, as it is estimated that data integrity management can represent up to 80% of the file system usage in PetaFlop data centers. As the emerging memory should become more dense and less expensive than DRAM, it will enable much greater capacity than main memories; this is especially important for big data applications, where storing large multidimensional tables in the main memory enables a tremendous performance advantage, because these tables often need to be accessed in a different order than the way it has been stored in the storage memory, which can only be accessed sequentially.
It will also simplify atomic operations in transactional databases, as RRAM are byte addressable and nonvolatile. They propose a much more efficient solution than current NVDIMMs composed of a mix of DRAM and flashes powered by a bulky supercapacitor during the data transfer from DRAM to Flash when the main power supply is lost. For these reasons, according to the specifications, fast (~100 ns) and high endurance (~10 9 cycles) RRAM and PCRAM could succeed. -It can also be used as a last-level cache replacement or complement; it is unlikely that memory technology could become fast enough to be used as a first-level cache. In this case, the cache capacity could be made much higher, diminishing the external bandwidth requirements. Here, the most stringent requirements would be speed (<30 ns) and endurance (>10 16 ).
We cannot rely on wear-leveling in this case as the cache capacity would not be sufficient in regard to its bandwidth, each bit being written frequently. A high endurance of 10 16 cycles is likely to reserve this application for STT-MRAM. FeRAM could be placed between an SCM memory (memory type) and the DRAM due to its high endurance. Finally, the only technology that exhibits performances close to SRAM is the SOT-MRAM. As far as cost and power consumption are concerned, the reference there is embedded DRAM (eDRAM); the RRAM has to be cheaper and less consuming than eDRAM to be competitive. A summary of the possible implementation of emerging memories in the memory hierarchy is presented in Figure  8. In this figure, storage class memory refers to a class of memory that stands between DRAM and disk storage in the data storage hierarchy.
Again, the gain in memory capacity has to be significant with respect to what can be achieved with SRAM. As endurance and capacity increase, the system efficiency and performance will be improved. Today, RRAM and PCRAM endurance is insufficient to perform computing in a device, and only specific computing tasks can be achieved with a limited number of cycles. On the contrary, MRAM appears to be a promising technology due to its excellent endurance. The challenge of this technology will be to improve its capacity, using more scalable spinbased concepts  [103] , in order to envisage the development of more complex and better performing systems. Ternary content addressable memory (TCAM) can also be listed as a computing architecture. TCAM performs parallel searches by comparing input searched data with data stored in the memory and returning the data address when a match occurs.
Spiking brain inspired neural network In spiking neural networks, the approach is to mimic human brain behavior and to represent data with spikes. Spiking neural networks (SNN) with spike-timing dependent plasticity (STDP) are then capable of replicating bio-realistic online/unsupervised learning  [120, 121] , which is not the case for deep learning architectures relying on supervised backpropagation. In SNN processes, the simultaneous spiking activity at two neurons can lead to a potentiation of the synapse connecting them, meaning that two neurons which are active in response to the same event, should be linked by a relatively strong synaptic connection. In this field, various demonstrations were proposed in the literature, aiming at implementing STDP in hardware synapses adopting various classes of emerging memories (including RRAM, CBRAM, STT-MRAM, and PCM).
Conclusions The general context of nonvolatile memories is characterized by several key points. First, the era of big data in which we live implies a constant and tremendous increase in data volume generation associated with the increase in the number of connected objects. The introduction into the market of 3D-NAND allowed pursuing a density increase. At the system level, the memory hierarchy suffers from two limitations, memory wall (between SRAM and DRAM) and latency gap (between DRAM and Flash), offering opportunities for new technologies. Data deluge also changed the paradigm of computing system; limited today by data transfer more than computing. This leads to the necessity for more efficient and specialized architectures, such as in-memory computing and neuromorphic circuits. At the component level, there is currently a renewed interest in emerging memory technologies (RRAM, PCRAM, MRAM, FeRAM, etc), based on "old" concepts due to the combination they offer in terms of fast speed, high endurance, and nonvolatility.
Zero-shot learning (ZSL) makes object recognition in images possible in absence of visual training data for a part of the classes from a dataset. When the number of classes is large, classes are usually represented by semantic class prototypes learned automatically from unannotated text collections. This typically leads to much lower performances than with manually designed semantic prototypes such as attributes. While most ZSL works focus on the visual aspect and reuse standard semantic prototypes learned from generic text collections, we focus on the problem of semantic class prototype design for large scale ZSL. More specifically, we investigate the use of noisy textual metadata associated to photos as text collections, as we hypothesize they are likely to provide more plausible semantic embeddings for visual classes if exploited appropriately.
Early works exploit manually created attributes  [39, 56, 29]  to define these prototypes. While very efficient, such attributes require a very costly annotation effort and are difficult to scale to large datasets. Different strategies were proposed to automate the creation of prototypes in order to tackle large scale ZSL. An early attempt  [47]  exploited WordNet to extract part attributes. While interesting, this method assumes that tested datasets can be mapped to WordNet, which is often impossible. The current trend, which leverages advances in natural language processing  [34, 21, 40] , is to exploit standard word embeddings as semantic prototypes. These embeddings are extracted from generic large scale text collections such as Wikipedia  [21, 34]  or Common Crawl  [6, 33]. The advantage of such methods is that prototype creation is based solely on webly supervised or unsupervised collections.
Results for a large scale collection show our approach enables consistent performance improvement compared to existing automatic prototypes. Interesting performance is also obtained for smaller datasets, where the proposed prototypes reduce the gap with manual prototypes. Our contributions can be summarized as follows: -We focus on the understudied problem of semantic prototype design for ZSL, and propose a method to create better embeddings from noisy tags datasets. -We conduct extensive experiments and ablation studies to  (1)  demonstrate the effectiveness of the proposed method; (2) provide a variety of results with different embeddings which can be used for future fair comparison;  (3)  provide insight on the remaining challenges to close the gap between manual and unsupervised semantic prototypes.
Recent generative approaches propose to learn discriminative models on unseen classes from artificial samples resulting from a generative model previously learned on seen classes  [54, 7, 53, 60]. The transductive ZSL setting assumes that the unlabelled visual testing samples can be used during training  [16, 26, 48, 52]. This usually boosts the performance, but we consider such a hypothesis too restrictive in practice, and this setting is out of the scope here. Semantic representation. Semantic prototypes can be created either manually or automatically. Since the former are difficult to scale, we focus on automatically created ones, that usually rely on large-scale datasets collected on the Web. The extraction of word representations from the contexts in which they appear is a longstanding topic in natural language processing (NLP).
It provides superior performance compared to continuous bag-of-words and skip gram models on a series of NLP tasks, including word analogy and similarity. The FastText model  [6]  derives from that proposed by Mikolov but considers a set of n-grams that can compose the words, compute some embeddings then represent a word as the sum of the vector representation of its n-grams. It thus models the internal structure of the words and thus allows to compute representations of out of vocabulary words. The state of the art in a large array of natural language processing task was recently improved by the introduction of contextual models such as ELMo  [41] , GPT  [43]  or BERT  [13]. These approaches make use of deep networks and model language at sentence level instead of word level as was the case for skip grams and GloVe.
However, ViCo is not usable in ZSL because it can only improve the representation of a word if some images of it are available. This drawback is inherent to all multimodal word representations and we thus focus on improving purely textual representations. Regarding visual features only,  [22, 55]  showed that one can train convolutional networks on a dataset of unannotated images collected on the Web, and that these networks perform well in a transfer learning context. Previous works in ZSL used embeddings to represent the semantic prototype, either at a small scale on CUB  [1]  or at a larger scale on ImageNet, using word2vec  [15, 11, 32]  (possibly trained on wikipedia  [9, 59] ) or GloVe  [10, 19]. However, they only use publicly available pre-trained models, while we propose a method to design prototypes that perform better in a ZSL context.
The default Flickr ranking algorithm is then used to collect up to 5000 photo metadata for each concept. Metadata fields which are exploited here include: (1) title -a free text description of the photo (2) tags -a list of tags attributed to the photo and (3) the unique identifier of the user. Note that there is no guarantee as to the relevance of textual metadata for the content of each photo since the users are free to upload any text they wish. Also, photo annotations can be made in any language. We illustrate title and tags from Flickr with the following examples:" Ísmáfur Pagophila eburnea Ivory Gull" and "minnesota flying inflight gull arctic juvenile duluth rare lakesuperior canalpark ivorygull saintlouiscounty". The title includes the Icelandic, Latin and English variants of the name while the tags give indications about the location and activity of the ivory gull.
Importantly, tags can be single words ("gull") or concatenated ones ("ivorygull","lakesuperior"). This first collection is made of 62. 7 million image metadata pieces and 1. 11 billion words. The f l wiki collection allows to learn generic embeddings that can be used to address large scale ZSL. However, these embeddings are still quite "generic" since they are representative of the Wikipedia concepts. For a given ZSL problem, the visual samples of unseen classes are unknown during training, but the name of these classes can be known before the actual production (testing) phase. Such a hypothesis is implicitly made by most generative ZSL approaches, which synthesize faked visual samples from the prototype only  [54, 7, 53, 60]. Following a similar hypothesis, we build fl cust , a custom subset of FlickR, which is built using the class names from the three ZSL used in evaluation datasets (ImageNet-ZSL, CUB and AWA).
Consequently, using a fixed size window to capture the context of a word is not suitable. We tested the use of fixed size windows in preliminary experiments and results were suboptimal. Instead, we form all distinct word pairs (v i , v j ), i = j, with v i , v j ∈ W n , for each piece of content associated to class c and feed them as training examples to the word embedding algorithms. Pairs extracted from all concept-related metadata collections M c = {m n } Nc n=1 associated with concept c are concatenated to form the training dataset of words embeddings, so that the objective becomes: C c=1 (vi,vj ), i =j vi,vj ∈{mn} Nc n=1 log p(v i |v j ) (4) Addressing repetitive tags. It is noteworthy that many users perform bulk tagging  [37]  which consists in attributing the same textual description to a whole set of photos.
Users also do semi-bulk, i. e. they attribute a part of tags to an entire photo set and then complete these annotations with photo-specific tags. Bulk is known to bias language models obtained from Flickr  [37, 42]  and we propose a simple but efficient way to remove it in the next section. To account for the bulk tagging problem, we add an additional processing step for the two collections. The authors of  [42]  and  [37]  suggested to replace simple tag co-occurrences by the number of distinct Flickr users who associated the two words and reported interesting gains in image retrieval and automatic geotagging respectively. We consequently select unique triplets (v i , v j , u n ) from M c for each concept c, so that training objective becomes C c=1 (vi,vj ,u k ), i =j vi,vj ,u k ∈Mc log p(v i |v j ) (5) This translates into adding a pair (v i , v j ) in the training file only once for each user and thus avoiding the effect of bulk tagging.
clue content is often made of short texts such as "ivory gull flying" which does not encode a lot of context. After sentence deduplication  [33] , the resulting collection includes 628 million unique metadata pieces and 3. 69 billion words. Evaluation datasets. The generic object recognition in ZSL requires to be evaluated at a large scale and is thus usually conducted on ImageNet  [12]. Frome et al. [15]  proposed to use the 1, 000 classes of ILSVRC for training and different subsets of the remaining 20, 841 classes to test. However, it has been recently showed that a structural bias appears in this setting which allows a "trivial model" to outperform most existing ZSL models  [19]. For this reason, we adopt the evaluation protocol proposed by Hascoet et al. that considers the same training classes as Frome et al.
but uses 500 classes with a minimal structural bias for testing  [19]. To get insight into the gap existing between manual attributes and unsupervised embeddings, we also conduct experiments on two smaller benchmarks on which the ZSL task is usually conducted with manual attributes specific to each dataset: Caltech UCSD Birds 200-2011 (CUB)  [56]  and Animals with Attributes 2 (AwA2)  [59]. CUB is a fine-grained dataset of 11788 pictures representing 200 bird species and AWA2 a coarse-grained dataset of 37322 pictures depicting 50 animal species. The manual attributes of CUB and AwA2 are respectively 312 and 85-dimensional. In our setting, we are only concerned with semantic prototypes which can be obtained automatically; our results therefore cannot be directly compared to the state-of-the-art algorithms which exploit manual attributes.
Such methods are outside the scope of this study. We additionally provide results for SynC  [9]  as well as two linear methods, consisting in a linear projection from the visual to the visual space (Linear V →S ), and a linear projection from the semantic to the visual space (Linear S→V ) inspired by  [50] , who proposed to compute similarities in the visual space to avoid the hubness problem  [45]. We train the models with the usual protocol for ZSL: hyperparameters are determined using a subset of training classes as validation. We sample respectively Table  1 : ZSL accuracy at large scale (ImageNet dataset), for three embedding models. Each time, the three baselines (pt, wiki and clue) are compared to our method fl wiki and its variation fl cust. Results marked with "*" correspond to a setting close to Table  2    Implementation details.
Comparison to other approaches The main results of the evaluation are reported in Table  1  for ImageNet. They confirm the relevance of our method and text collections to learn semantic prototypes for ZSL, as the best results are consistently obtained with our prototypes. Specifically, for ImageNet, the best result reported on the unbiased split in  [19]  is 14. 1 with ADGPM, and 13. 5 with a "traditional" ZSL model (not making use of additional nodes in the class hierarchy), which used GloVe embeddings pretrained on Common Crawl. By contrast, our best result is 17. 2 with FastText, obtained with embeddings trained on a much smaller dataset. We also provide results for CUB and AwA2 in Table  2. These results are less relevant since manual attributes exist for these smaller scale datasets, but still provide interesting insights.
Importantly, these results are obtained using unsupervised prototypes and should not be directly compared to results obtained with manual attributes. On CUB, the best results are obtained with the embeddings learned on the f l cust collection for the three configurations and significantly outperform previous embeddings. Interestingly, there does not seem to be a clear tendancy on AwA2. It turns out that performance obtainable with unsupervised prototypes on AwA2 is already quite close to performance with manual attributes -see Sec. 4. 4. Our method is therefore unable to provide a significant improvement, unlike on the other two datasets. Within each embeddings methods for all three datasets, the best results are usually obtained with f l cust and f l wiki usually performs better than baseline methods.
The gain is especially large when compared to the largest available pretrained models for word2vec and FastText. This result is obtained although the largest collections used to create pretrained embeddings are 2 to 3 orders of magnitude larger than the collections we use. For GloVe on ImageNet, the model pretrained on Common Crawl has the best performance. This embedding has poor behavior for all smaller scale datasets, indicating that the combination of local and global contexts at its core is able to capture interesting information at large scale. While its performance on the smaller pretrained dataset is significantly lower than that of FastText, the two models are nearly equivalent when trained on Common Crawl. A similar finding was reported for text classification tasks  [33]. The strong performance of f l cust follows intuition since the collec- tion was specifically built to cover the concepts which appear in the three test dataset.
This finding confirms the usefulness of smaller but adapted collections for NLP applications such as medical entity recognition  [14]  or sentiment analysis  [23]. Note that we also combined f l wiki and f l cust to obtain a more generic Flickr model. The obtained results were only marginally better compared to the single use of f l cust and are reported in the supplementary material. We note that FastText and word2vec are better than GloVe for ImageNet and CUB, the two largest datasets with respect to number of classes. Overall, the best performance is usually obtained with f l cust and FastText embeddings.
Comparison to manual attributes Although our webly semantic prototypes enable to achieve much better results than with previously available prototypes extracted from text corpora, it is still Fig. 1 : Ablation of manual attributes on CUB and AwA2. Measured with the linear model, averaged over 10 runs. Best results for prototypes based on word embeddings are also reported. interesting to compare them to what can be achieved with hand-crafted attributes. Such attributes do not exist for very large scale datasets such as Im-ageNet, but they are provided with smaller scale datasets such as CUB and AwA2. To quantify how much better hand-crafted prototypes perform when compared to webly supervised prototypes, we conducted an ablation study on CUB attributes similar to Sec. 4. 3. We started with the full list of attributes, initially comprising 312 attributes for each bird species, and randomly removed attributes while measuring the resulting ZSL score.