dataesr/scientific-paragraphs-categorization huggingface dataset:
Surface Water Area Dynamics Surface water is easily detected and classified using optical and radar remote sensing. In the present context of impermanent lakes composing the WHS, the main refilling period occurs during the monsoon, when cloud cover is thick and permanent. Sentinel-1 was preferred for its ability to detect surface roughness through cloud cover. The temporal dataset is automatically used to extract surface water areas for each acquisition. The areas estimated for each surface reservoir are a crucial variable in estimating the role of evaporation and infiltration fluxes taking place below the WHS. This is a first step in quantifying the potential contribution of WHS to regional aquifer recharge by capturing monsoon runoff  [15, 26, 39].
Effect of Spatial Resolution The 10-m spatial resolution did not dramatically improve the classification scores but doubled the size of the dataset (58 and 26 Gb for Sentinel-1 and 2, respectively, at 10 and 20 m in Rabi 2016, Table  3 ). For larger areas, we find the 20-m resolution more appropriate since it provides similar area estimates for rice, more robust results for irrigated crops, and smaller datasets.
Conclusions Flooded rice areas and surface water dynamics are essential variables in predicting groundwater usage in a small South Indian watershed. The first Sentinel-1 and -2 time series are used to estimate the accuracy of retrieval of irrigated and flooded sown crop areas for three contrasted season: the dry Rabi 2016 (R2016, December 2015 to March 2016), the monsoon season, known as Kharif 2016 (K2016, July to November 2016) and the next Rabi season in 2017 (R2017, December 2016 to March 2017). The spatial resolution at 10 or 20 m and the types of sensors are combined into six datasets to explore the retrieval accuracy of each variable. Small irrigated areas are detected during the dry conditions of R2016 and K2016, as 3. 5% and 5%, respectively, of the irrigated areas, with moderate classification confusion.
A large increase of irrigated areas (16% of rice and 6% of irrigated crops) is estimated with low classification confusion in R2017. This is explained by the replenishment of water resources during the 2016 monsoon. This high seasonal variability represents the direct impact of the farmers' strategy of adaptation to water storage. In Rabi 2017, Sentinel-1 only, Sentinel-2 only and Sentinel-1 and 2 datasets show good classification accuracy (F-score for rice above 0. 90) although Sentinel-1 used alone leads to a 10% overestimation of the rice areas (lower precision at 10-m resolution). In contrast the Kharif cotton areas seem to be accurately detected by Sentinel-1 only; their estimated extent remains similar (around 20%) after introducing Sentinel-2 (only four cloud-free images during the Kharif).
The value of 20% appears highly realistic, as a similar area was found for Kharif 2009  [17, 18]. Sentinel-1 and 2 in Kharif and both Sentinel-1 and 2 and Sentinel-2 alone in Rabi seem to give the best estimates. We combined the results from these two datasets at both 10 and 20 m resolution to compute the water demand uncertainty for each season: 49. 5 ± 0. 78 mm (1. 5% uncertainty) in Rabi 2016, 44. 9 ± 2. 9 mm (6. 5% uncertainty) in the Kharif season and 226. 2 ± 5. 8 mm (2. 5% uncertainty) in Rabi 2017. These amounts should be compared to seasonal aquifer recharge variation over the past decade (10 to 160 mm/year). The confusion over detection of irrigated crops in small areas in Rabi 2016 leads to a small uncertainty in water requirement estimates. The 20-m resolution should be selected when scaling this methodology up to larger areas.
Acknowledgments: This work was carried out in the context of the project " Agro-hydrology from Space ," co-funded by the  Living Planet fellowship program of the European Spatial Agency (ESA)  and the research program BAG'AGES funded by the  Agence de l'Eau Adour-Garonne (AEAG). The study was directed by Sylvain Ferrant thanks to the collaboration of the  Indo-French Center for Groundwater Research (IFCGR). The authors would like to thank the IFCGR field team, especially  Wajiduddin , and  Pierre Pottin  and  Bianca Hoersch , respectively Sentinel-1 and 2 mission managers from  European Space Agency  for their help at the beginning and throughout the project. We would like to thank  JohnV. Guy-Bray  for the English revision.
FraFigure 7. 7 Figure 7. Seasonal land cover derived from Sentinel-1 and 2 datasets at 20 m over the Kudaliar watershed. The dataset is especially consistent for rice and irrigated crops. High confusion remains between bare ground, bush, and urban land covers. Surface water appears during the monsoon season (Kharif 2016) and its extent starts to decrease during the following dry season (Rabi 2017).
We introduce GuessWhat?!, a two-player guessing game as a testbed for research on the interplay of computer vision and dialogue systems. The goal of the game is to locate an unknown object in a rich image scene by asking a sequence of questions. Higher-level image understanding, like spatial reasoning and language grounding, is required to solve the proposed task. Our key contribution is the collection of a large-scale dataset consisting of 150K human-played games with a total of 800K visual question-answer pairs on 66K images. We explain our design decisions in collecting the dataset and introduce the oracle and questioner tasks that are associated with the two players of the game. We prototyped deep learning models to establish initial baselines of the introduced tasks.
Yes No No Yes The last few years has seen an increasing interest from the computer vision community in tasks towards this goal. Thanks to advances in training deep neural networks  [16]  and the availability of large-scale classification datasets [26,  35, 49] , automatic object recognition has now reached human-level performance  [24]. As a result, attention has been shifted toward tasks involving higher-level image understanding. One prominent example is image captioning  [26] , the task of automatically producing natural lan- guage descriptions of an image. Visual Question Answering (VQA)  [6]  is another popular task that involves answering single open-ended questions concerning an image. Closer to our work, the ReferIt game  [21]  aims to generate a single expression that refers to one object in the image.
On the other hand, there has been a renewed interest in dialogue systems  [31, 37] , inspired by the success of datadriven approaches in other areas of natural language processing  [11]. Traditionally, dialogue systems have been built through heavy engineering and hand-crafted expert knowledge, despite machine learning attempts for almost two decades  [25, 40]. One of the difficulties comes from the lack of automatic evaluation as -contrary to machine translation -there is no evaluation metric that correlates well with human evaluation  [27]. A promising alternative is goal-directed dialogue tasks  [31, 40, 44, 43]  where agents converse to pursue a goal rather than casually chit-chat. The agent's success rate in completing the task can then be used as an automatic evaluation metric. Many tasks have recently been introduced, including the bAbI tasks  [44]  for testing an agent's ability to answer questions about a short story, the movie dialog dataset  [12]  to assess an agent's capabilities regarding personal movie recommendation and a Wizardof-Oz framework  [43]  to evaluate an agent's performance for assisting users in finding restaurants.
In this paper, we bring these two fields together and propose a novel goal-directed task for multi-modal dialogue. The two-player game, called GuessWhat?!, extends the ReferIt game  [21]  to a dialogue setting. To succeed, both players must understand the relations between objects and how they are expressed in natural language. From a machine learning point of view, the GuessWhat?! challenge is the following: learn to acquire natural language by interaction on a visual task. Previous attempts in that direction  [2, 43]  do not ground natural language to their immediate environment; instead they rely on an external database through which a conversational agent searches. The key contribution of this paper is the introduction of the GuessWhat?! dataset that contains 155,280 dialogues composed of 831,889 question/answer pairs on 66,537 images extracted from the MS COCO dataset  [26].
We define three sub-tasks that are based on the GuessWhat?! dataset and prototype deep learning baselines to establish their difficulty. The paper is organized as follows. First, we explain the rules of the GuessWhat?! game in Sec. 2. Then, Sec. 3 describes how GuessWhat?! relates to previous work. In Sec. 4. 1 we highlight our design decisions in collecting the dataset, while Sec. 4. 2 analyses many aspects of the dataset. Sec. 5 introduces the questioner and oracle tasks and their baseline models. Finally, Sec. 6 provides a final discussion of the GuessWhat?! game.
Related work The GuessWhat?! game and the data collected from it present opportunities for the extension of current research on image captioning, visual question answering and dialogue systems. In the following, we describe previous work in these areas and relate them to the open challenges offered by GuessWhat?!. We also mention other relevant work on dataset collection. Image captioning Our work builds on top of the MS COCO dataset  [26]  which consists of 120k images with more than 800k object segmentations. In addition, the dataset provides 5 captions per image which initiated an explosion of interest from the research community into generating natural language descriptions of images. Several methods have been proposed  [20, 42, 45] , all inspired by the encoder-decoder approach  [11, 41]  that has proven successful for machine translation.
Image captioning research uncovered successful approaches to automatically generate coherent, factual statements about images. Modeling the interactions in GuessWhat?! requires instead to model the process of asking useful questions about images. VQA datasets Visual Question Answering (VQA) tasks form another well known extension of the captioning task. They instead require answering a question given a picture (e. g. "How many zebras are there in the picture?", "Is it raining outside?" ). Recently, the VQA challenge  [6]  has provided a new dataset far bigger than previous attempts  [15, 29]  where, much like in GuessWhat?!, questions are free-form. An extensive body of work has followed from this publication, largely building on the image captioning literature  [3, 28, 39, 46]. Unfortunately, many of these advanced methods were shown to marginally improve on simple baselines  [19].
Recent work  [3]  also reports that trained models often report the same answer to a question irrespective of the image, suggesting that they largely exploit predictive correlations between questions and answers present in the dataset. The GuessWhat?! game and dataset attempt to circumvent these issues. Because of the questioner's aim to locate the hidden object, the generated questions are different in nature: they naturally favour spatial understanding of the scene and the attributes of the objects within it, making it more valuable to consult the image. Besides, it only contains binary questions, whose answers we find to be balanced and has twice more questions on average per picture. Goal-directed dialogue GuessWhat?! is also relevant to the goal-directed dialogue research community. Such systems are aimed at collaboratively achieving a goal with a user, such as retrieving information or solving a problem.
Although goal-directed dialogue systems are appealing, they remain hard to design. Thus, they are usually restricted to specific domains such as train ticket sales, tourist information or call routing  [32, 40, 47]. Besides, existing dialogue datasets are either limited to fewer than 100k example dialogues  [12] , unless they are generated with template formats  [12, 43, 44]  or simulation  [33, 36]  in which case they don't reflect the free-form of natural conversations. Finally, recent work on end-to-end dialogue systems fail to handle dynamical contexts. For instance,  [43]  intersects a dialogue with an external database to recommend restaurants. Well-known game-based dialogue systems [1, 2] also rely on static databases. In contrast, Guess-What?! dialogues are heavily grounded by the images.
The resulting dialogue is highly contextual and must be based on the content of the current picture rather than an external database. Thus, to the best of our knowledge, the Guess-What?! dataset marks an important step for dialogue research, as it is the first large scale dataset that is both goaloriented and multi-modal. Human computation games GuessWhat?! is in line with Von Ahn's seminal work on human computation games  [4, 5]  who showed that games are an effective way to gather labeled data. The first ESP game [4] was developed to collect image tags, and was later extended to Peekaboom  [5]  to gather object segmentations. These games were developed more than a decade ago, when object recognition was in its infancy and served a different purpose than GuessWhat?!. ReferIt Probably closest to our work is the ReferIt game  [21, 30, 48].
In this game, one player observes an annotated object in a scene, for which they need to generate an expression that refers to it (e. g. ẗhe man wearing the white t-shirt¨). The other player then receives this expression and subsequently clicks on the location of the object within the image. The original dataset  [21]  uses the IMAGEClef dataset  [13] , while three recent extensions  [30, 48]  were built on top of MS COCO. All three databases select images with only 2 -4 objects of the same category. In contrast, GuessWhat?! picks images with 3 -20 objects without further restrictions on the object class, and thus contains three times more images than the ReferIt datasets. To further investigate the difference between ReferIt and GuessWhat?!, we compare three samples for the same selected object in Fig  14 in Appendix B.
Data collection Images We use a subset of the training and validation images and objects of the MS COCO dataset  [26]. We first discard objects that are too small (area < 500px 2 ) to be decently located by a human observer. Then, we only keep images containing three to twenty objects, to avoid trivial or overly complicated images. In total, we keep 77,973 images with 609,543 objects. We verified that this selection does not significantly alter the original dataset distribution.
In the end, we only kept dialogues from qualified people and successful dialogues from the qualification round. In contrast to traditional dataset collection, our game requires an interactive session between two players. Fortunately, we found that the GuessWhat?! game was highly engaging. A total of more than 10K people participated in our HITs, and our top ten participants played over 2, 000 games each. Since questions were manually typed, they could contain spelling mistakes. Thus, we retrieved all questions containing words that do not occur in an English dictionary and manually corrected the 1000 most common words. For the remaining 30k questions, we created two HITs that to correct the spelling mistakes. See Figure  10.
Dataset statistics The raw GuessWhat?! dataset is composed of 155,280 dialogues containing 821,889 question/answer pairs on 66,537 unique images and 134,073 unique objects. The answers are respectively 52. 2% no, 45. 6% yes and 2. 2% N/A. On average, there are 5. 2 questions per dialogue and 2. 3 dialogues per image. The dialogues contain 3,986,192 word tokens in total, making up 11,465 different words with at least one occurrence and 5,444 words with at least 3 occurrences. Moreover, 84. 6% of the dialogues are successful, 8. 4% are unsuccessful and 7. 0% are not completed (disconnection, timeout etc. ). Thus, different subsets co-exist in the GuessWhat?! dataset, we will refer to the dataset as full, finished and successful when we include all the dialogues, all finished dialogues (successful and unsuccessful) or only successful dialogues, respectively.
For more details, the previous statistics are broken down into dataset types in Tab 1. Question distributions To get a better understanding of the GuessWhat?! games, we show the number of questions within a dialogue and the average number of questions given the number of objects within a image in Fig 3. First, the number of questions within a dialogue decreases exponentially, as players tend to shorten their dialogues to speed up the game (and therefore maximize their gains). More interestingly, we observe that the average number of questions given the number of objects within an image appears to follow a function that grows at a rate between logarithmically and linearly. A questioning strategy of simply listing objects (e. g. "is it the chair", etc. ) would imply linear growth in the number of questions, while the optimal binary search strategy would imply logarithmic growth.
Oracle baselines The oracle task requires to produce a yes-no answer for any object within a picture given a natural language question. We first introduce our model and then outline its results to get a better understanding of the GuessWhat?! dataset. Model We propose a simple neural network based approach to this model, illustrated in Fig 5. Specifically, we use an appropriate neural network architecture to embed each of the following information: the image I, the cropped object from S, its spatial information, its category c and the current question q. These embeddings are then concatenated as a single vector and fed as input to a single hidden layer MLP that outputs the final answer distribution using a softmax layer. Finally, we minimize the cross-entropy error during the training and report the classification error at evaluation time.
We normalize the image height and width such that coordinates range from -1 to 1, and place the origin at the center of the image. As for the object category, we convert its one-hot class vector into a dense category embedding using a learned look-up table. Finally, the embedding of the current natural language question q is computed using an Long Short-Term Memory (LSTM) network  [17]  where questions are first tokenized by using the word punct tokenizer from the python nltk toolkit  [7]. For simplicity, we decided to ignore the question-answer pairs history q <t in our oracle baseline. Training setting We train all oracle models on the full dataset. During training, we keep the parameters of the VGG network fixed, and optimize the LSTM, object category/word look-up tables and MLP parameters by minimizing the negative log-likelihood of the correct answer.
We use ADAM  [22]  for optimization and train for at most 15 epochs. We use early stopping on the validation set, and report the train, valid and test error. Results We report results for several oracle models using a different set of inputs in  model after the input we feed to it. For instance, (Ques-tion+Category+Spatial+Image) refers to the network fed with the question q, the object category c, the spatial features x spatial and the full image I. The results of all subsets are reported in Table  6  in Appendix C. Because the GuessWhat?! dataset is fairly balanced, simply outputting the most common answer in the training set -No -results in a high 50. 8% error rate. Solely providing the image or crop features barely improves upon this result. Only using the question slightly improves the error rate to 41.
We encode the image by extracting its FC8 features from VGG16 network. A dialogue of a GuessWhat?! game is a sequence on two different levels: there is a variable number of question-answer pairs where each question in turn consists of a variable-length sequence of tokens. This can be encoded into a fixed size vector by using either an LSTM encoder  [17]  or an HRED encoder  [38]. While the LSTM encoder considers the dialogue as one flat sequence, HRED explicitly models the hierarchy by two different Recurrent Neural Networks (RNN). First, an encoder RNN creates a fixed-size representation of a question or answer by reading in its tokens and taking the last hidden state of the RNN. This representation is then processed by the context RNN to obtain a representation of the current dialogue state.
Discussion We introduced the GuessWhat?! game, a novel framework for multi-modal dialogue. To the best of our knowledge, we present the first large-scale dataset involving images and dialogue. A wide range of challenges may arise from this union as they rely on different fields of machine learning such as natural language understanding, generative models or computer vision. GuessWhat?! turns out to be an engaging game that greatly decreases the cost for collec-tion of a big dataset required for modern algorithms. As a second contribution, we introduced three tasks based on the questioner and oracle role. In each case, we prototyped a neural architecture as a first baseline. We analyzed these results and presented a quantitative description of the Guess-What?! dataset. We believe GuessWhat?! could allow for a myriad of other applications that may either be based on the game itself or extending the database to other tasks.
For instance, it can be interesting to compute a confidence interval before proceeding to the final guess. Differently, GuessWhat?! could be a test bed for one-shot learning  [14]  of guessing new object categories, transfer learning on line-drawing images [10] or using questions from another language. Thus, the GuessWhat?! dataset offers an opportunity to develop original machine learning tasks upon it. Figure  15  presents a word matrix of the GuessWhat?! dataset. Figure  16  and Figure  17a  compares the object size and category distribution of GuessWhat?! with MS Coco.
Acknowledgement The authors would like to acknowledge the stimulating environment provided by the  MILA and SequeL labs. We thank all members of the  MILA lab  who participated in a trial run of the data collection, and all workers of AMT who participated in our HITs. We thank  Jake Snell ,  Mengye Ren ,  Laurent Dinh ,  Jeremie Mary  and  Bilal Piot  for helpful discussions. We acknowledge the following agencies for research funding and computing support:  NSERC ,  Calcul Québec, Compute Canada , the  Canada Research Chairs  and  CIFAR ,  CHISTERA IGLU  and  CPER Nord-Pas de Calais/FEDER   DATA Advanced data science  and technologies 2015-2020. SC is supported by a  FQRNT-PBEEE scholarship.
Relative evolution of topics during a dialogue of size 6. We applied Data Topic Models (DTM) [8]  with the python framework [34]  on our dataset. The table reports the two prominent detected topics with their respective key words while the figure display their relative evolution during the dialogue. The topic titles are manually picked. 1. 0 0. 8 0. 6 0. 4 0. 2 0. 0 Success Failure Incomplete Topic 1 Abstract words Descriptive words Topic 2 person left food one vehicle right human wearing car side one white object red animal table Figure 19: carrot broccoli book banana orange traffic light sheep apple boat spoon potted plant car bird donut handbag knife skis cow bottle bench chair scissors snowboard bicycle toothbrush vase oven hot dog bowl truck backpack wine glass cup keyboard suitcase parking meter cake couch sandwich motorcycle fork remote person microwave kite Proportion of topics toaster 0.
Data Collection Procedures The researchers held a meeting with possible volunteers at the institution. They were explained the goals and ethical procedures and gave their informed consent. Data collection was performed by trained evaluators at two different moments  [21]. On the first day the individuals responded to the instruments (STAI, POMS, SF-36 and BAECKE) and on the second day anthropometric variables were collected (Weight and Height). All individuals received the same verbal instructions and any doubts were clarified before the questionnaires were filled out. The instruments also contained written instructions on how to complete them. During the application of the instruments, there was no communication between individuals and there was no time limit imposed  [21].