NCube/papyrus huggingface dataset:
Compression in Sequence to Sequence Learning for Natural Language Processing

In this work, we propose a near lossless method for encoding long sequences of texts as well as all of their sub-sequences into feature rich representations. We test our method on sentiment analysis and show good performance across all sub-sentence and sentence embeddings. This work also demonstrates the use of knowledge distillation and quantization to compress the original Transformer model [Vaswani et al., 2017] for the translation task. We are, to the best of our knowledge, the first to show that 8-bit quantization of the weights of the Transformer can achieve the same BLEU score as the full-precision model. Furthermore, when we combine knowledge distillation with weight quantization, we can train smaller Transformer networks and achieve up to 12.59x compression while losing only 2.51 BLEU off the baseline on the WMT 2014 English-to-French translation task. Chapter 1 introduces machine learning concepts for natural language processing which are essential to understanding both papers presented in this thesis. Chapter 2 and 3 cover each paper respectively, before finally concluding with chapter 4.

Keyphrases: compression, machine learning, deep learning, natural language processing, sequence to sequence
On the bias-variance tradeoff : textbooks need an update

The main goal of this thesis is to point out that the bias-variance tradeoff is not always true (e.g. in neural networks). We advocate for this lack of universality to be acknowledged in textbooks and taught in introductory courses that cover the tradeoff.  We first review the history of the bias-variance tradeoff, its prevalence in textbooks, and some of the main claims made about the bias-variance tradeoff. Through extensive experiments and analysis, we show a lack of a bias-variance tradeoff in neural networks when increasing network width. Our findings seem to contradict the claims of the landmark work by Geman et al. (1992). Motivated by this contradiction, we revisit the experimental measurements in Geman et al. (1992). We discuss that there was never strong evidence for a tradeoff in neural networks when varying the number of parameters. We observe a similar phenomenon beyond supervised learning, with a set of deep reinforcement learning experiments.  We argue that textbook and lecture revisions are in order to convey this nuanced modern understanding of the bias-variance tradeoff.

Keyphrases: Bias-variance tradeoff, Neural networks, Over-parameterization, Generalization
Towards learning sentence representation with self-supervision

In chapter 1, we introduce the basics of feed forward neural networks and recurrent neural networks. The chapter continues with the discussion of the backpropagation algorithm to train feed forward neural networks, and the backpropagation through time algorithm to train recurrent neural networks. We also discuss three different approaches in learning representations, namely supervised learning, unsupervised learning, and a relatively new approach called self-supervised learning.  In chapter 2, we talk about the fundamentals of deep natural language processing. Specifically, we cover word representations, sentence representations, and language modelling. We focus on the evaluation and current state of the literature for these concepts. We close the chapter by discussing large scale pre-training and transfer learning in language.  In chapter 3, we investigate a set of self-supervised tasks that take advantage of noise contrastive estimation in order to learn sentence representations using unlabeled data. We train our model on a large corpora and evaluate our learned sentence representations on a set of downstream natural language tasks from the SentEval framework. Our model trained on the proposed tasks outperforms unsupervised methods on a subset of tasks from SentEval.  In chapter 4, we introduce a memory augmented model called Ordered Memory with several improvements over traditional stack-augmented recurrent neural networks. We introduce a new Stick-breaking attention mechanism inspired by Ordered Neurons [Shen et.al., 2019] to write in and erase from the memory. A new Gated Recursive Cell is also introduced to compose low level representations into higher level ones. We show that this model performs well on the logical inference task and the ListOps task, and it also shows strong generalization properties in these tasks. Finally, we evaluate our model on the SST (Stanford Sentiment Treebank) tasks (binary and fine-grained) and report results that are comparable with state-of-the-art on these tasks.

Keyphrases: Neural networks, Deep learning, Representation learning, Unsupervised learning, Natural language processing, Language modelling, Memory augmented neural networks
Towards Understanding Generalization in Gradient-Based Meta-Learning

In this master's thesis, we study the generalization of neural networks in gradient-based meta-learning by analyzing various properties of the objective landscapes. Meta-learning, a challenging paradigm where models not only have to learn a task but beyond that, are trained for ``learning to learn" as they must adapt to new tasks and environments with very limited data about them. With research on the objective landscapes of neural networks in classical supervised having provided some answers regarding their ability to generalize for new data points, we propose similar analyses aimed at understanding generalization in meta-learning. We first introduce the literature on objective landscapes of neural networks in Section \ref{sec:intro:objective_landscapes}. We then introduce the literature of meta-learning in Section \ref{chap:prof_forcing}, concluding our introduction with the approach of gradient-based meta-learning, a meta-learning setup that bears strong similarities to the traditional supervised learning setup through stochastic gradient-based optimization. At the time of writing of this thesis, and to the best of our knowledge, this is the first work to empirically study the objective landscapes in gradient-based meta-learning, especially in the context of deep learning. We notably provide some insights on some properties of those landscapes that appear correlated to the generalization to new tasks. We experimentally demonstrate that as meta-training progresses, the meta-test solutions, obtained after adapting the meta-train solution of the model, to new tasks via few steps of gradient-based fine-tuning, become flatter, lower in loss, and further away from the meta-train solution. We also show that those meta-test solutions become flatter even as generalization starts to degrade, thus providing experimental evidence against the correlation between generalization and flat minima in the paradigm of gradient-based meta-leaning. Furthermore, we provide empirical evidence that generalization to new tasks is correlated with the coherence between their adaptation trajectories in parameter space, measured by the average cosine similarity between task-specific trajectory directions, starting from a same meta-train solution. We also show that coherence of meta-test gradients, measured by the average inner product between the task-specific gradient vectors evaluated at meta-train solution, is also correlated with generalization. Based on these observations, we propose a novel regularizer for the Model Agnostic Meta-Learning (MAML) algorithm and provide experimental evidence for its effectiveness.

Keyphrases: Deep learning, Meta-learning, Objective landscapes, Generalization, Few-shot learning
Loghorrée, aphasie, lettres : les états de la parole dans le cinéma d’Arnaud Desplechin

This present study’s aim is suggesting a linguistic analysis throughout different scenes in a medley of four films by the french director Arnaud Desplechin: Comment je me suis disputé… (« ma vie sexuelle ») (1996); Rois et reine (2004); Un conte de Noël (Roubaix!) (2008) et Trois souvenirs de ma jeunesse (nos arcadies) (2015). First of all, it looks into the qualifications of long periods of time granted to a very talkative character. The definition of soliloquy, such as described previously cannot be applied to these long speeches since they cannot be included in the dialogue logic. There is almost no real oral exchange. Communication cannot be established. Then this study will also analyze the different soliloques and their influence on the narration, according to the character’s physical or mental state together with his credibility. His disability to talk depends on different causes for the film maker : sickness, insomnia, aphonia, death, intoxication, drugs, neurosis, even psychosis. After exploring the different restrictions to the character’s communication, this essay will shortly focus on the comical side effects of these speeches. Then we will examine the special importance of letters which are essential in this film corpus. This essay also deals with the writing of these important words. They are separated from the rest of the story. They have a different temporality, their speech delivery is close to theatrical reciting. The production chooses filming front the front, explaining the reasons of rising the means of correspondance, pondering over the danger of silence which is, at times, either undergone or induced by the character through the speech. Since this analysis concerns the linguistic writing, both oral and filmed, the complexity of filming speech appears. As a conclusion, numerous new paths need to be explored when studying film making as far as filming speech is concerned.

Keyphrases: Arnaud Desplechin, Mathieu Amalric, French cinema, Words, Speech, Soluloquy, Long speeches, Letters, Correspondance, Neurosis
La mémoire et son paysage minéral : des images stratigraphiques explorées par la parole

This film research-creation project focuses on the history of a former mining town. In 1971, a landslide triggered by the excavations of the adjacent mine swept part of the town of Asbestos – and its history – to the bottom of the pit. This symbolic incident serves as the basis for an exploration of how best to represent collective memory in film. In eschewing the archival approach of documentary cinema (using images and audio recordings from the past), I have tried to show what has disappeared from the town’s physical landscape yet is still very present in the memory of its inhabitants. My project is underpinned by three main questions: 1) Can Asbestos’s past be evoked through images of its current landscape? 2) Could capturing the spoken words of its inhabitants be the best way to bring a community’s experiences back to life and share them with others? 3) Can a bygone past be revealed through the interplay of present images and vernacular accounts of recent or distant history?

Keyphrases: mémoire, mine, strates, Asbestos, Memory, History, Community, Orality, Speech, Layer, Geology
Le centenaire de la Grande Guerre au prisme des médias : le cas des quotidiens The Guardian, Le Monde et die Süddeutsche Zeitung

This mémoire aims to show that media space, in times of commemoration, is a privileged place for historians who want to capture the dynamics of a community’s memory. Our case study is the media coverage of the centenary of the First World War through the lens of The Guardian, Le Monde and Die Süeddeutsche Zeitung. We analysed in these newspapers the press coverage from June 2014 to July 2017 in the online sections First world war, Centenaire 14-18 and Erster Weltkrieg. This study uses an operative definition of collective memory that allows us to understand the necessity, in the history of memory field, to study journalism not simply as an archive, but also as a discursive space where various complex representations of the past meet and communicate in order to redefine themselves. This discursive space has two points of junction: Practiced history and “mythistory”. The dynamics of this space have been studied for each publication in this research. First, we identify the major historical configurations of the Great War, the role of historians in the coverage, and try to see to which extent these configurations are expressed in the three newspapers studied. Finally, the commemorative intentions of the newspapers are identified and then tested through a study of news-flow and an analysis of the lexical fields of the respective media coverage.

Keyphrases: Centenaire, France, Centenary, Great War, First World War, journalism, remembrance, mediatic space, collective memory, France, United Kingdom, Germany
Gigantisme naval et ports d'escale : un nouveau paradigme

The growth in the size of container vessels and their deployment is a reality. Currently, the growth of container ships is at 21 413 TEUs, and the order books for ever larger vessels portend an even greater trend towards larger vessels. For instance, CMA CGM will be delivered nine new 23 000 TEU container ships by the end of 2020. The objective of this paper is to characterize the movement of ultra-large container vessels (ULCV) on several shipping routes, with different carriers and regions of the world. This objective is two-fold; the first part deals with the variability of the dwell time of ULCV in the ports, and the second deals with the planning of ULCV routes for each maritime carrier. Historical data of the movement of a large sample of ULCV was analysed with a view to address a series of key issues. Here are some of these issues: What are the mains ports of call? What is the variability of dwell time between the ports of call? What are the routing patterns of mega containers vessels? This study will allow us to differentiate productivity of different ports of call, regions and shipping carriers on many shipping lanes.

Keyphrases: Service maritime, Megaships, ULCV, Container, Port of call, Dwell time, Turnaround time, Transshipment, Maritime services
Espèces végétales non-natives sur une île subantarctique chilienne : étude des perceptions des parties prenantes, modélisation des dynamiques de niches des espèces et cartographie des points chauds

The growth of human activities over the last century has contributed to the dispersal and establishment of non-native plant species around the world. These species are important drivers of environmental change since they have the capacity to modify native plant communities, especially through competition. Since human are the main drivers of introduction, establishment and spread of non-native species and since management strategies need to take into account public attitudes to be successful, it is necessary to understand the perceptions and choices regarding their use and management. The present study aimed to: i) understand the perceptions of different stakeholders towards the different non-native plant species of Navarino Island, a buffer zone of the Cape Horn Biosphere Reserve (CHBR) in the Chilean sub-Antarctic region and ii) identify niche dynamics and high potential establishment areas (i.e. hotspots) of four of these species, which together will serve as multidisciplinary tools for the CHBR management. To address the first objective, seven stakeholder groups were identified (i.e residents, elder residents, conservationists, marine members, tourists, Yaghan community members, public service employees), within which 21 semi-directed interviews were conducted. To address the second objective, global and regional occurrences of four of the most abundant species, i.e. Bellis perennis, Cerastium fontanum, Taraxacum officinale and Trifolium repens and associated climatic data were used to develop species distribution models using the maximum entropy method (MaxENT). Regarding the first objective, results of this study revealed shortcomings in the non-native concept understanding among stakeholders in a local context. On the other hand, the analyzes revealed many positive values associated with non-native plant species in Navarino, notably the aesthetic value. Nevertheless, this study showed that most stakeholders were indifferent to the management of non-native herbaceous plant species. Regarding the second objective, results revealed a niche shift usually used by species due to the less favorable climate of Navarino Island as well as high plasticity for the four plant species mentioned above. However, plant populations might have reached the equilibrium. Finally, the combination of global and regional models shows that hotspots areas are located more particularly on the south coast of the island and extend to the north depending on the species. Based on the results obtained in this study, we therefore suggest (i) increasing awareness of non-native plant among stakeholders, (ii) integrating their values into future decisions, and (iii) preventing dispersal of non-native plant species through Antarctica by monitoring species distribution on Navarino.

Keyphrases: perceptions, conservation, Exotic species, Modelling, Niche, Biodiversity, Protected areas, Sub-Antarctic
Discours idéologiques et violence de masse : la légitimation de la destruction en contexte colonial

This study is about the role of ideology in mass violence within colonial context. In the 19th century, counter-insurrection campaigns aiming to put down colonial revolts and mutinies against Empires have often targeted civilians. In India, during the Indian Mutiny of 1857, thousands of non-combatants have been killed in answer to the insurrection. In this study, we look at the role of ideology in that kind of colonial atrocities. Theoretically, we try to clarify the link between ideology and violent practices on the ground. We argue that ideological analysis can gain explanatory force by putting aside, without refuting it, the idea of ideology as a motivation to kill. Instead, we conceptualize it as a collective phenomenon that engages the legitimation of violence within armed groups. Empirically, a case study in a colonial context is proposed from Leader Maynard's theory of the ideological mechanisms that can legitimize violence. The discourse analysis of soldiers on the ground during the Indian Mutiny has permitted to identify some ideological particularities that can enrich this theory. We point out a tension in the colonial discourse between the supposed benefits of colonialism brought to the colonial subjects, and the need to make sense of their destruction in a context of insurrection. In summary, this analysis draws a portrait of the different discursive forms related to violence, while clarifying their role in these destructive dynamics leading to the indiscriminate killings of noncombatants.

Keyphrases: Discours, Ideology, Discourse, Legitimacy, Mass violence, Colonialism
La plainte du chien battu, ou la poésie désenchanteresse (1991-2013) de Michel Houellebecq

This master’s dissertation is a study of Michel Houellebecq’s poetry according to the theoretical framework of sociocritic (Duchet, Robin, Popovic). Urban life, social life and alterities of the city are the most present themes in the poetical work of the author. The main hypothesis is that the writing mobilizes and take over signs, linguistic acts and representations which float around in the French contemporary society, but in a dynamic way: the scriptural act of writing modifies their meaning and criticizes their common use. In the first chapter, we will discuss the “putting into text” (literal translation of the « mise en texte » concept by Duchet, which is the action of putting something into the form of a literary text) of urban life, which shines a light on how much the economic laws rule the city and its inhabitants. It describes a market world where money reigns. In the second chapter, the “putting into text” of social life and people-to-people ties shows how surprisingly artificial they are and certifies that their organization shrinks individual liberties more and more. Having free time to spend with your loved ones was once a given of social modernity but is now becoming difficult to manage. In the third and last chapter, the putting into text of the othernesses of the city (suburbs, province, nature) reveals that they offer no alternative to the hostility of the urban world and the fact that it is, and it will always be, every man for himself. Despite this distressing diagnosis, poetry tries to detect here and there hints of hope and solidarity.

Keyphrases: Michel Houellebecq, French contemporary literature, poetry, sociocritic, social and cultural representations (city, social life, suburb, nature, party, crowd, work, spare time)
Les pratiques de gestion à haute performance et l'épuisement émotionnel

This thesis aims to explore the indirect effect of high-performance work practices on emotional exhaustion. Job resources (job control and supervisors social support) and job demands (psychological demands and work hours) are theorized as mediation mechanisms in this relationship. Job demands-resources theory (Demerouti et.al, 2001) in background of this work is enhanced by the addition of a group-level construct in the explanation of emotional exhaustion. Therefore, this study aims to test two antagonistic perspectives on the effect of high-performance work practices on occupational health. The data for this study is from the SALVEO study conducted between 2009 and 2012 by the Équipe de recherche sur le travail et la santé mentale (ERTSM). A total of 1,961 employees in 89 employee groups nested in 60 workplaces were surveyed on their perceived level of job control, supervisor social support, psychological demands, work hours and emotional exhaustion. The line managers were surveyed on the implementation of high-performance work practices. To test hypotheses, descriptive analysis, bivariate analysis, and multilevel path analysis were performed. Results suggest that job resources reduce emotional exhaustion whereas psychological demands increase it. The finding further reject the pluralist perspective and confirm the unitarist approach by arguing that some high-performance management practices such as compensation and participation in decision-making are associated with less emotional exhaustion by providing more job resources to individuals. Others like staffing are also associated with less emotional exhaustion by reducing jobs demands. Moreover, we did not find evidence that combination of high-performance management practices had more effect on individuals than individual practices. Results highlight the importance of continuing studies on the association between human resources management and occupational health.

Keyphrases: High-performance work practices, Job demands, Job resources, Emotional exhaustion, Occupation health