OpenRaiser/Intern-Atlas huggingface dataset:
Learning Music Helps You Read: Using Transfer to Study Linguistic Structure in Language Models
Venue: arXiv cs.CL 2020
Year: 2020

We propose transfer learning as a method for analyzing the encoding of grammatical structure in neural language models. We train LSTMs on non-linguistic data and evaluate their performance on natural language to assess which kinds of data induce generalizable structural features that LSTMs can use for natural language. We find that training on non-linguistic data with latent structure (MIDI music or Java code) improves test performance on natural language, despite no overlap in surface form or vocabulary. To pinpoint the kinds of abstract structure that models may be encoding to lead to this improvement, we run similar experiments with two artificial parentheses languages: one which has a hierarchical recursive structure, and a control which has paired tokens but no recursion. Surprisingly, training a model on either of these artificial languages leads to the same substantial gains when testing on natural language. Further experiments on transfer between natural languages controlling for vocabulary overlap show that zero-shot performance on a test language is highly correlated with typological syntactic similarity to the training language, suggesting that representations induced by pre-training correspond to the cross-linguistic syntactic properties. Our results provide insights into the ways that neural models represent abstract syntactic structure, and also about the kind of structural inductive biases which allow for natural language acquisition.
Can Your Context-Aware MT System Pass the DiP Benchmark Tests? : Evaluation Benchmarks for Discourse Phenomena in Machine Translation
Venue: arXiv cs.CL 2020
Year: 2020

Despite increasing instances of machine translation (MT) systems including contextual information, the evidence for translation quality improvement is sparse, especially for discourse phenomena. Popular metrics like BLEU are not expressive or sensitive enough to capture quality improvements or drops that are minor in size but significant in perception. We introduce the first of their kind MT benchmark datasets that aim to track and hail improvements across four main discourse phenomena: anaphora, lexical consistency, coherence and readability, and discourse connective translation. We also introduce evaluation methods for these tasks, and evaluate several baseline MT systems on the curated datasets. Surprisingly, we find that existing context-aware models do not improve discourse-related translations consistently across languages and phenomena.
Standard words and solutions of the word equation $X_1^2 \dotsm X_n^2 = (X_1 \dotsm X_n)^2$
Venue: arXiv cs.FL 2020
Year: 2020

We consider solutions of the word equation $X_1^2 \dotsm X_n^2 = (X_1 \dotsm X_n)^2$ such that the squares $X_i^2$ are minimal squares found in optimal squareful infinite words. We apply a method developed by the second author for studying word equations and prove that there are exactly two families of solutions: reversed standard words and words obtained from reversed standard words by a simple substitution scheme. A particular and remarkable consequence is that a word $w$ is a standard word if and only if its reversal is a solution to the word equation and $\gcd(|w|, |w|_1) = 1$. This result can be interpreted as a yet another characterization for standard Sturmian words. We apply our results to the symbolic square root map $\sqrt{\cdot}$ studied by the first author and M. A. Whiteland. We prove that if the language of a minimal subshift $Ω$ contains infinitely many solutions to the word equation, then either $Ω$ is Sturmian and $\sqrt{\cdot}$-invariant or $Ω$ is a so-called SL-subshift and not $\sqrt{\cdot}$-invariant. This result is progress towards proving the conjecture that a minimal and $\sqrt{\cdot}$-invariant subshift is necessarily Sturmian.
Attentive Weakly Supervised land cover mapping for object-based satellite image time series data with spatial interpretation
Venue: arXiv cs.CV 2020
Year: 2020

Nowadays, modern Earth Observation systems continuously collect massive amounts of satellite information. The unprecedented possibility to acquire high resolution Satellite Image Time Series (SITS) data (series of images with high revisit time period on the same geographical area) is opening new opportunities to monitor the different aspects of the Earth Surface but, at the same time, it is raising up new challenges in term of suitable methods to analyze and exploit such huge amount of rich and complex image data. One of the main task associated to SITS data analysis is related to land cover mapping where satellite data are exploited via learning methods to recover the Earth Surface status aka the corresponding land cover classes. Due to operational constraints, the collected label information, on which machine learning strategies are trained, is often limited in volume and obtained at coarse granularity carrying out inexact and weak knowledge that can affect the whole process. To cope with such issues, in the context of object-based SITS land cover mapping, we propose a new deep learning framework, named TASSEL (aTtentive weAkly Supervised Satellite image time sEries cLassifier), that is able to intelligently exploit the weak supervision provided by the coarse granularity labels. Furthermore, our framework also produces an additional side-information that supports the model interpretability with the aim to make the black box gray. Such side-information allows to associate spatial interpretation to the model decision via visual inspection.
Sim-to-Real Transfer with Incremental Environment Complexity for Reinforcement Learning of Depth-Based Robot Navigation
Venue: arXiv cs.RO 2020
Year: 2020

Transferring learning-based models to the real world remains one of the hardest problems in model-free control theory. Due to the cost of data collection on a real robot and the limited sample efficiency of Deep Reinforcement Learning algorithms, models are usually trained in a simulator which theoretically provides an infinite amount of data. Despite offering unbounded trial and error runs, the reality gap between simulation and the physical world brings little guarantee about the policy behavior in real operation. Depending on the problem, expensive real fine-tuning and/or a complex domain randomization strategy may be required to produce a relevant policy. In this paper, a Soft-Actor Critic (SAC) training strategy using incremental environment complexity is proposed to drastically reduce the need for additional training in the real world. The application addressed is depth-based mapless navigation, where a mobile robot should reach a given waypoint in a cluttered environment with no prior mapping information. Experimental results in simulated and real environments are presented to assess quantitatively the efficiency of the proposed approach, which demonstrated a success rate twice higher than a naive strategy.
Binary autoencoder with random binary weights
Venue: arXiv cs.LG 2020
Year: 2020

Here is presented an analysis of an autoencoder with binary activations $\{0, 1\}$ and binary $\{0, 1\}$ random weights. Such set up puts this model at the intersection of different fields: neuroscience, information theory, sparse coding, and machine learning. It is shown that the sparse activation of the hidden layer arises naturally in order to preserve information between layers. Furthermore, with a large enough hidden layer, it is possible to get zero reconstruction error for any input just by varying the thresholds of neurons. The model preserves the similarity of inputs at the hidden layer that is maximal for the dense hidden layer activation. By analyzing the mutual information between layers it is shown that the difference between sparse and dense representations is related to a memory-computation trade-off. The model is similar to an olfactory perception system of a fruit fly, and the presented theoretical results give useful insights toward understanding more complex neural networks.
Observed mobility behavior data reveal social distancing inertia
Venue: arXiv cs.CY 2020
Year: 2020

The research team has utilized an integrated dataset, consisting of anonymized location data, COVID-19 case data, and census population information, to study the impact of COVID-19 on human mobility. The study revealed that statistics related to social distancing, namely trip rate, miles traveled per person, and percentage of population staying at home have all showed an unexpected trend, which we named social distancing inertia. The trends showed that as soon as COVID-19 cases were observed, the statistics started improving, regardless of government actions. This suggests that a portion of population who could and were willing to practice social distancing voluntarily and naturally reacted to the emergence of COVID-19 cases. However, after about two weeks, the statistics saturated and stopped improving, despite the continuous rise in COVID-19 cases. The study suggests that there is a natural behavior inertia toward social distancing, which puts a limit on the extent of improvement in the social-distancing-related statistics. The national data showed that the inertia phenomenon is universal, happening in all the U.S. states and for all the studied statistics. The U.S. states showed a synchronized trend, regardless of the timeline of their statewide COVID-19 case spreads or government orders.
Hierarchical clustering of bipartite data sets based on the statistical significance of coincidences
Venue: arXiv cs.SI 2020
Year: 2020

When some 'entities' are related by the 'features' they share they are amenable to a bipartite network representation. Plant-pollinator ecological communities, co-authorship of scientific papers, customers and purchases, or answers in a poll, are but a few examples. Analyzing clustering of such entities in the network is a useful tool with applications in many fields, like internet technology, recommender systems, or detection of diseases. The algorithms most widely applied to find clusters in bipartite networks are variants of modularity optimization. Here we provide an hierarchical clustering algorithm based on a dissimilarity between entities that quantifies the probability that the features shared by two entities is due to mere chance. The algorithm performance is $O(n^2)$ when applied to a set of n entities, and its outcome is a dendrogram exhibiting the connections of those entities. Through the introduction of a 'susceptibility' measure we can provide an 'optimal' choice for the clustering as well as quantify its quality. The dendrogram reveals further useful structural information though -- like the existence of sub-clusters within clusters or of nodes that do not fit in any cluster. We illustrate the algorithm by applying it first to a set of synthetic networks, and then to a selection of examples. We also illustrate how to transform our algorithm into a valid alternative for one-mode networks as well, and show that it performs at least as well as the standard, modularity-based algorithms -- with a higher numerical performance. We provide an implementation of the algorithm in Python freely accessible from GitHub.
Vocabulary Adaptation for Distant Domain Adaptation in Neural Machine Translation
Venue: arXiv cs.CL 2020
Year: 2020

Neural network methods exhibit strong performance only in a few resource-rich domains. Practitioners, therefore, employ domain adaptation from resource-rich domains that are, in most cases, distant from the target domain. Domain adaptation between distant domains (e.g., movie subtitles and research papers), however, cannot be performed effectively due to mismatches in vocabulary; it will encounter many domain-specific words (e.g., "angstrom") and words whose meanings shift across domains(e.g., "conductor"). In this study, aiming to solve these vocabulary mismatches in domain adaptation for neural machine translation (NMT), we propose vocabulary adaptation, a simple method for effective fine-tuning that adapts embedding layers in a given pre-trained NMT model to the target domain. Prior to fine-tuning, our method replaces the embedding layers of the NMT model by projecting general word embeddings induced from monolingual data in a target domain onto a source-domain embedding space. Experimental results indicate that our method improves the performance of conventional fine-tuning by 3.86 and 3.28 BLEU points in En-Ja and De-En translation, respectively.
On the Baldwin Effect under Coevolution
Venue: arXiv cs.NE 2020
Year: 2020

The potentially beneficial interaction between learning and evolution, the Baldwin effect, has long been established. This paper considers their interaction within a coevolutionary scenario, ie, where the adaptations of one species typically affects the fitness of others. Using the NKCS model, which allows the systematic exploration of the effects of fitness landscape size, ruggedness, and degree of coupling, it is shown how the amount of learning and the relative rate of evolution can alter behaviour.
Non-Volatile Kernel Root kit Detection and Prevention in Cloud Computing
Venue: arXiv cs.CR 2020
Year: 2020

The field of web has turned into a basic part in everyday life. Security in the web has dependably been a significant issue. Malware is utilized to rupture into the objective framework. There are various kinds of malwares, for example, infection, worms, rootkits, trojan pony, ransomware, etc. Each malware has its own way to deal with influence the objective framework in various ways, in this manner making hurt the framework. The rootkit may be in some arbitrary records, which when opened can change or erase the substance or information in the objective framework. Likewise, by opening the rootkit contaminated record may debase the framework execution. Hence, in this paper, a Kernel Rootkit Detection and Prevention (KRDP) framework is proposed an avert the records. The avoidance system in this paper utilizes a calculation to forestall the opening of the rootkit influenced record as portrayed. By and large, the framework comprises of a free antivirus programming which is restricted to certain functionalities. The proposed model beats the functionalities by utilizing a calculation, in this way identifying the rootkits first and afterward cautioning the client to react to the rootkit tainted record. In this way, keeping the client from opening the rootkit contaminated record. Inevitably, in the wake of expelling the tainted document from the framework will give an improvement in the general framework execution.
Mutlitask Learning for Cross-Lingual Transfer of Semantic Dependencies
Venue: arXiv cs.CL 2020
Year: 2020

We describe a method for developing broad-coverage semantic dependency parsers for languages for which no semantically annotated resource is available. We leverage a multitask learning framework coupled with an annotation projection method. We transfer supervised semantic dependency parse annotations from a rich-resource language to a low-resource language through parallel data, and train a semantic parser on projected data. We make use of supervised syntactic parsing as an auxiliary task in a multitask learning framework, and show that with different multitask learning settings, we consistently improve over the single-task baseline. In the setting in which English is the source, and Czech is the target language, our best multitask model improves the labeled F1 score over the single-task baseline by 1.8 in the in-domain SemEval data (Oepen et al., 2015), as well as 2.5 in the out-of-domain test set. Moreover, we observe that syntactic and semantic dependency direction match is an important factor in improving the results.
Using Decision Diagrams to Compactly Represent the State Space for Explicit Model Checking
Venue: arXiv cs.SE 2020
Year: 2020

The enormous number of states reachable during explicit model checking is the main bottleneck for scalability. This paper presents approaches of using decision diagrams to represent very large state space compactly and efficiently. This is possible for asynchronous systems as two system states connected by a transition often share many same local portions. Using decision diagrams can significantly reduce memory demand by not using memory to store the redundant information among different states. This paper considers multi-value decision diagrams for this purpose. Additionally, a technique to reduce the runtime overhead of using these diagrams is also described. Experimental results and comparison with the state compression method as implemented in the model checker SPIN show that the approaches presented in this paper are memory efficient for storing large state space with acceptable runtime overhead.
Aspect-Controlled Neural Argument Generation
Venue: arXiv cs.CL 2020
Year: 2020

We rely on arguments in our daily lives to deliver our opinions and base them on evidence, making them more convincing in turn. However, finding and formulating arguments can be challenging. In this work, we train a language model for argument generation that can be controlled on a fine-grained level to generate sentence-level arguments for a given topic, stance, and aspect. We define argument aspect detection as a necessary method to allow this fine-granular control and crowdsource a dataset with 5,032 arguments annotated with aspects. Our evaluation shows that our generation model is able to generate high-quality, aspect-specific arguments. Moreover, these arguments can be used to improve the performance of stance detection models via data augmentation and to generate counter-arguments. We publish all datasets and code to fine-tune the language model.
Leveraging Context-awareness to Better Support the IoT Cloud-Edge Continuum
Venue: arXiv cs.NI 2020
Year: 2020

Novel Internet of Things (IoT) requirements derived from a broader interconnection of heterogeneous devices have pushed the horizons of Cloud computing and are giving rise to a wider decentralisation of applications and data centers. An answer to the underlying network concerns, such as the need to lower the resulting latency due to heavy computation needs,or safety aspects, gave rise to Edge/Fog computing, where IoT functionality can be also supported closer to data sources. While it is today feasible to perform some IoT functionality on the Edge, the orchestration of operations between Edge and Cloud requires an automated support, where context-awareness plays a key role in assisting the network in deciding when and whereto store data and to perform computation. This work is focused on the application of context-awareness to support a smoother operation of the Edge to Cloud operation, aiming at lowering latency, in particular when real-time or close-to-real-time data exchange is present.
Conceptual Design of Human-Drone Communication in Collaborative Environments
Venue: arXiv cs.HC 2020
Year: 2020

Autonomous robots and drones will work collaboratively and cooperatively in tomorrow's industry and agriculture. Before this becomes a reality, some form of standardised communication between man and machine must be established that specifically facilitates communication between autonomous machines and both trained and untrained human actors in the working environment. We present preliminary results on a human-drone and a drone-human language situated in the agricultural industry where interactions with trained and untrained workers and visitors can be expected. We present basic visual indicators enhanced with flight patterns for drone-human interaction and human signaling based on aircraft marshaling for humane-drone interaction. We discuss preliminary results on image recognition and future work.
Three Levels of Modeling: Static (Structure/Trajectories of Flow), Dynamic (Events) and Behavioral (Chronology of Events)
Venue: arXiv cs.SE 2020
Year: 2020

Constructing a conceptual model as an abstract representation of a portion of the real world involves capturing the (1) static (things/objects and trajectories of flow), (2) the dynamic (event identification), and (3) the behavior (e.g., acceptable chronology of events) of the modeled system. This paper focuses on examining the behavior notion in modeling and current works in the behavior space to illustrate that the problem of behavior and its related concepts in modeling lacks a clear-cut systematic basis. The purpose is to advance the understanding of system behavior to avoid ambiguity-related problems in system specification. It is proposed to base the notion of behavior on a new conceptual model, called the thinging machine, which is a tool for modeling that establishes three levels of representation: (1) a static structural description that is constructed upon the flow of things in five generic operations (activities; i.e., create, process, release, transfer and receive); (2) a dynamic representation that identifies hierarchies of events based on five generic events; and (3) a chronology of events. This is shown through examples that support the thinging machine as a new methodology suitable for all three levels of specification.
On the Merging of Domain-Specific Heterogeneous Ontologies using Wordnet and Web Pattern-based Queries
Venue: arXiv cs.IR 2020
Year: 2020

Ontologies form the basic interest in various computer science disciplines such as semantic web, information retrieval, database design, etc. They aim at providing a formal, explicit and shared conceptualization and understanding of common domains between different communities. In addition, they allow for concepts and their constraints of a specific domain to be explicitly defined. However, the distributed nature of ontology development and the differences in viewpoints of the ontology engineers have resulted in the so called "semantic heterogeneity" between ontologies. Semantic heterogeneity constitutes the major obstacle against achieving interoperability between ontologies. To overcome this obstacle, we present a multi-purpose framework which exploits the WordNet generic knowledge base for: i) Discovering and correcting the incorrect semantic relations between the concepts of the ontology in a specific domain. This step is a primary step of ontology merging. ii) Merging domain-specific ontologies through computing semantic relations between their concepts. iii) Handling the issue of missing concepts in WordNet through the acquisition of statistical information on the Web. And iv) Enriching WordNet with these missing concepts. An experimental instantiation of the framework and comparisons with state-of-the-art syntactic and semantic-based systems validate our proposal.
Why and when should you pool? Analyzing Pooling in Recurrent Architectures
Venue: arXiv cs.CL 2020
Year: 2020

Pooling-based recurrent neural architectures consistently outperform their counterparts without pooling. However, the reasons for their enhanced performance are largely unexamined. In this work, we examine three commonly used pooling techniques (mean-pooling, max-pooling, and attention), and propose max-attention, a novel variant that effectively captures interactions among predictive tokens in a sentence. We find that pooling-based architectures substantially differ from their non-pooling equivalents in their learning ability and positional biases--which elucidate their performance benefits. By analyzing the gradient propagation, we discover that pooling facilitates better gradient flow compared to BiLSTMs. Further, we expose how BiLSTMs are positionally biased towards tokens in the beginning and the end of a sequence. Pooling alleviates such biases. Consequently, we identify settings where pooling offers large benefits: (i) in low resource scenarios, and (ii) when important words lie towards the middle of the sentence. Among the pooling techniques studied, max-attention is the most effective, resulting in significant performance gains on several text classification tasks.