dataesr/scientific-paragraphs-categorization huggingface dataset:
Feature Processing We converted raw features into a form suitable for modeling using a fve-stage pipeline. First, we apply one-hot encoding to categorical features. Second, we set numeric values above the 99th percentile or below the 1st percentile to those respective cut-ofs. Third, we imputed missing categorical values using the mode of non-missing values, and missing numeric values with the mean of non-missing values. Fourth, we removed the mean of numeric felds and scaled to unit variance. Lastly, we randomly removed datasets that were exact deduplicates of each other, resulting in unique 1, 066, 443 datasets and 2, 884, 437 columns. However, many datasets are slight modifcations of each other, uploaded by the same user. Therefore, we removed all but one randomly selected dataset per user, which also removed bias towards more prolifc Plotly users.
Prediction Tasks Our task is to train models that use the features described in Section 4 to predict the design choices also described in Section 4. Two visualization-level prediction tasks use dataset-level features to predict visualization-level design choices: The three encoding-level prediction tasks use features about individual columns to predict how they are visually encoded. These prediction tasks consider each column independently, instead of alongside other columns in the same dataset, which accounts for the efect of column order. ( For the Visualization Type and Mark Type tasks, the 2class task predicts line vs. bar, and the 3-class predicts scatter vs. line vs. bar. Though Plotly supports over twenty mark types, we limited prediction outcomes to the few types that comprise the majority of visualizations within our corpus.
Neural Network and Baseline Models Our primary model is a fully-connected feedforward neural network (NN) with 3 hidden layers, each consisting of 1, 000 neurons with ReLU activation functions and implemented using PyTorch  [41]. For comparison, we chose four simpler baseline models, all implemented using scikit-learn  [42]  with default parameters: naive Bayes (NB), K-nearest neighbors (KNN), logistic regression (LR) and random forest (RF). Randomized parameter search for each model did not result in a signifcant performance increase over the reported results. For all models, we split the data into 60/20/20 train/validation/test sets and train and test each model fve times using 5-fold cross-validation. The reported results are thus test results averaged across the fve test sets. We oversample the train, validation, and test sets to the size of the majority class while ensuring no overlap between the three sets.
The dependence on number of columns is related to the heuristics described by Bertin  [5]  and encoded in Show Me  [34]. Features related to column type ( ) are consistently important for each prediction task. For example, whether a dataset contains a string type column is the ffth most important feature for determining two-class visualization type. The dependence of visualization type choice on column data type is consistent with the type-dependency of the perceptual properties of visual encodings described by Mackinlay  [35]  and Cleveland and McGill  [15]. ) such as Gini, entropy, skewness and kurtosis are important across the board. The presence of these higher order moments is striking because lower-order moments such as mean and variance are low in importance. The importance of these moments highlight the potential importance of capturing high-level characteristics of distributional shape.
These observations support the use of statistical properties in visualization recommendation, like in  [59, 70] , but also the use of higher-order properties such as skewness, kurtosis, and entropy in systems such as Foresight  [16] , VizDeck  [43] , and Draco  [37]. Measures of orderedness ( ), specifcally sortedness and monotonicity, are important for many tasks. Sortedness ), name ( ), and pairwise-relationship ( ). is defned as the element-wise correlation between the sorted and unsorted values of a column, that is |corr(X r aw , X sor t ed )|, which lies in the range [0, 1]. Monotonicity is determined by strictly increasing or decreasing values in X r aw. The importance of these features could be due to pre-sorting of a dataset by an analyst, which may reveal which column is considered to be the independent or explanatory column, which is typically visualized along the X-axis.
Modeling and Measuring Efectiveness As discussed in Section 2, we model data visualization as a process of making a set of design choices C = {c} that maximize an efectiveness criteria Ef that depends on dataset d, task, and context. In Section 6, we predict these design choices by training a machine learning model on a corpus of dataset-design choice pairs [(d, c d )]. But because each dataset was visualized only once by each user, we consider the user choices c d to be efective, and each other choice as inefective. That is, we consider efectiveness to be binary. But prior research suggests that efectiveness is continuous. For example, Saket et al. use time and accuracy preference to measure task performance  [53] , Borkin et al. use a normalized memorability score  [6] , and Cleveland and McGill use absolute error rates to measure performance on elementary perceptual tasks  [15].
Discussions by visualization experts  [25, 29]  also suggest that multiple visualizations can be equally efective at displaying the same data. Our efectiveness metric should be continuous and refect the ambiguous nature of data visualization, which leads to multiple choices receiving a non-zero or even maximal score for the same dataset. This is in agreement with measures of performance for other machine learning tasks such as the BLEU score in language translation  [40]  and the ROUGE metric in text summarization  [11] , where multiple results can be (partly) correct. To estimate this efectiveness function, we need to observe a dataset d visualized by multiple potential users. Assume that a design choice c can take on multiple discrete values {v}. For instance, we consider c the choice of Visualization Type, which can take on the values {bar, line, scatter}.
Second, it lets us test whether models trained on the Plotly corpus can generalize and if Plotly users are actually making optimal choices. Lastly, it lets us benchmark against performance of the Plotly users as well as other predictors. To generate the crowdsourced evaluation data, we recruited and successfully pre-screened 300 participants through Amazon Mechanical Turk. The data preparation and crowdsourced evaluation procedures is described in more detail in SM Section S6.
The two learning-based predictors are DeepEye and Data2Vis. In all cases, we tried to make choices that maximize prediction performance, within reason. We uploaded datasets to Show Me, DeepEye, and CompassQL as comma-separated values (CSV) fles, and to Data2Vis as JSON objects. Unlike VizML and Data2Vis, DeepEye supports pie, bar, and scatter visualization types. We marked both pie and bar recommendations were both bar predictions, and scatter recommendations as line predictions in the two-type case. For all tools, we modifed the data within reason to maximize the number of valid results. For the remaining errors (4 for Data2Vis, 14 for DeepEye), and cases without returned results (12 for DeepEye and 33 for CompassQL) we assigned a random chart prediction. Predictor performance is evaluated as the total sum of normalized efectiveness scores.
This Consensus-Adjusted Recommendation Score (CARS) of a predictor is defned as: � 1 Õ P ˆ c cˆp r edict or, d CARS pr edict or = × 100 (1) |D| max ({P ˆ c }) d ∈D where |D| is the number of datasets (66 for two-class and 99 for three-class), cˆp r edict or, d is the predicted visualization type for dataset d, and P ˆ c returns the fraction of Mechanical Turker votes for a given visualization type. Note that the minimum CARS > 0%. We establish 95% confdence intervals around these scores by comparing against 10 5 bootstrap samples of the votes, which can be thought of as synthetic votes drawn from the observed probability distribution.
DISCUSSION In this paper, we introduce VizML, a machine learning approach to visualization recommendation using a large corpus of datasets and corresponding visualizations. We identify fve key prediction tasks and show that neural network classifers attain high test accuracies on these tasks, relative to both random guessing and simpler classifers. We also benchmark with a test set established through crowdsourced consensus, and show that the performance of neural networks is comparable that of individual humans. Visualization system developers have multiple paths towards incorporating ML-based recommenders such as VizML into authoring workfows. Partial specifcation recommenders on top of existing manual specifcation tools, such as the Show Me  [34]  feature in Tableau  [62] , rely on design choice suggestions that could be provided by a learned model.
Alternatively, developers can overcome the cold-start problem by using pre-trained models such as VizML. With models in hand, developers can progress further by collecting the usage analytics (e. g. measures of engagement such as clicks and shares) to establish customized measures of visualization efectiveness. We acknowledge the limitations of the Plotly corpus and our approach. First, despite aggressive deduplication, our model is certainly biased towards the Plotly dataset. As a web-based platform, Plotly could draw a certain cohort of analysts, encourage certain types of plots by interface design or defaults, or be more appropriate for specifc types and sizes of data. Second, neither the Plotly user nor the Mechanical Turker is an expert in data visualization. Thirdly, we acknowledge that this paper was only focused on a subset of the tasks usually considered in a visualization recommendation pipeline.
Promising avenues for future work lie in both data collection and modelling directions. On the data side, there is a need for more diverse training data from other tools (e. g. Many Eyes and Tableau) and pertaining to adjacent data science tasks such as feature selection and data transformation. Richer training data allows researchers to investigate the previous bias concerns, optimize visualization recommenders with a task-based (or generally multi-objective) efectiveness metric, recommend multiple views of a dataset, study complementary approaches to feature engineering, and integrate distinct design choice recommendations using a probabilistic graphical model. Underlying each ML-based recommender model is a measure of visualization efectiveness. Determining the parameters that inform efectiveness is an open question for the visualization community.
Machine learning tasks such as image annotation or medical diagnosis are often objective, in that there exists a clear human-annotated ground truth. Other tasks are subjective, such as language translation or text summarization tasks, and are benchmarked against human evaluation or against human-generated results. Questions of objective visualization quality point towards the role of experts in visualization assessment. Visualization experts provide evaluations that are informed by experience and knowledge of perceptual studies. But if laypeople are the target audience of visualizations, the consensus opinion of crowdsourced agents may be a good measure of visualization quality. By providing a large training corpus, initial machine learning models, and a crowdsourced benchmark, VizML is a step forward in addressing these questions.
In many countries, commodities provided by the agriculture sector play an important role in the economy. Securing food is one aspect of this role, which can be achieved when the decision makers are supported by tools. The need for cheap, fast, and accurate tools with high temporal resolution and global coverage has encouraged the decision makers to use remote sensing technologies. Field spectroradiometer with high spectral resolution can substantially improve crop mapping by reducing similarities between different crop types that have similar ecological conditions. This is done by recording fine details of the crop interaction with sunlight. These details can increase the same crop recognition even with the variation in the crop chemistry and structure. This paper presents a new spectral signatures database interactive tool (CSSIT) for the major crops in the Eastern Mediterranean Basin such as wheat and potato.
The creation of a full database of crop spectral signatures is a complex task because it requires continuous monitoring and observations of the crops from seeding to harvesting. Currently, the existing spectral signature libraries are mostly made for non-crops analysis and research purposes. These libraries such as the Johns Hopkins University (JHU) Spectral Library  [4] , the Jet Propulsion Laboratory (JPL) Spectral Library  [5] , the United States Geological Survey (USGS-Reston) Spectral Library  [6] , the ASTER spectral library version 2. 0  [7] , and the ASU Spectral Library  [8]  include only spectral signatures of rocks, minerals, lunar soils, terrestrial soils, manmade materials, meteorites, some North American vegetation, snow, and ice. There are other spectral signatures libraries designed for agriculture research purposes such as the SPECCHIO Spectral online database maintained by the Remote Sensing Laboratories in the Department of Geography at the University of Zurich  [1, 9].
This database displays metadata about instrument characteristics, date of acquisition, vegetation biophysical parameters, soil characteristics, and other important information. Moreover, the tool can display the spectral signature of major crops such as wheat and potato. The library has been tested, but there are many problems, such as no multi criteria query. A long list is displayed for each search which sometimes consists of hundreds of records. Instead, it should be more specific to the date and if possible to the location too. It should also specify more information about the crop itself for example plantation date of the crop. There are more libraries and databases for crop, vegetation and agriculture purpose, but some are no longer in service, such as the Vegetation Spectral Library, which was developed by the Systems Ecology Laboratory at the University of Texas in El Paso with support from the National Science Foundation  [10].
The above reasons promoted the idea for developing a new crop spectral signatures database to improve crop management and to help establish precision in agricultural practices. In addition to the spectral signatures, the database includes a list of crop characteristics, by which a specific crop may be identified on an image. One of these characteristics is the Chlorophyll-a estimation which can help in the precise estimation of nitrogen fertilizer use as part of precision agriculture  [11]. Another characteristic which can help in establishing precision in agriculture is to estimate plant water content using the library and remote sensing images. It may also provide valuable information to environmental and irrigation system managers to relieve dehydration symptoms and prevent permanent growth and production damage  [12].
A new spectral signatures database is most useful if it is implemented as part of a software application that includes an attractive, effective, and friendly graphic user interface (GUI). The tool can be deployed to map crops accurately. Normally, crop mapping requires tedious and continuous field work to verify and classify the satellite hyperspectral images. In the case of crop mapping, field work can become intolerable and expensive when crop classification requires images for different dates concurrently with crop growth stages. This is sometimes a necessary step in order to avoid differences in the date of plantation between different farmers. The problem can be solved if spectral signatures for crops at different growth phases are collected. Several important issues must be considered during the collection of data for building the spectral signature database such as the exact number of in situ measurements for obtaining the correct crop spectral signature and the sufficient number of times needed to collect data for each crop in different phenological stages.
One of the possible uses of the parameters is to create a mathematical model for every major crop to predict these parameters from remote sensing data such as the one created for the chlorophyll-a content for wheat crop in  [16]. In this research, an interactive tool (CSSIT) is developed. It is planned to include a spectral signatures database of major crops with physiochemical parameters. The tool will also include metadata for each survey (environmental conditions and others) and multi criteria search, retrieve, display, and analysis of the crop spectral signatures.
Data Description To succeed in creating the new tool, several important issues must be considered. First we have to decide on the type of data to be collected such as spectral signatures, leaf area index, and chlorophyll-a content. The collection of these data is based on using an ASD FieldSpec 4 Hi-Res Spectroradiometer (Figure  1a )  [17] , SPAD 502 (Figure  1b )  [18] , and LI-3000C Portable Leaf Area Meter (Figure  1c )  [19]. The 3 nm visible and near infrared (VNIR), 8 nm short wave infrared (SWIR) spectral resolution of the FieldSpec 4 Hi-Res spectroradiometer provides superior spectral performance across the full range of the solar irradiance spectrum (350-2500 nm). Combining spectral signatures, biophysical, and biochemical parameters can support modeling efforts by providing spatially and temporally distributed information on important vegetation characteristics, which would be very difficult to obtain otherwise.
Regarding the time of data acquisition, the field spectroscopy protocols  [23]  recommend that the measurement time must be as close as possible to solar noon. In our case, most of the field work is done between 10:30 in the morning and 2:30 in the afternoon. It is important that in order to create unique and measureable spectral signatures for crops, spectral signatures databases must consider the spatiotemporal changeability of the vegetation. Field surveys should be done on different phenological cycles for each crop and to detect differences in LAI, CHC, and other physiochemical parameters that can help distinguish between different crops and to create unique signature for specific crop. In our case and depending on the climate conditions, the trips are conducted once every week such that each week a different area is visited and surveyed.
Methods The creation of the tool requires analysis and design of user requirements. In other words, organized and planned steps are conducted to create the database management system (DBMS) and link it to thematic functions of the tool such as displaying spectral signature graph, calculation of vegetation indices, resampling of spectral signatures, and showing the biophysical parameters. This includes a user friendly interface with easy to use dialog boxes in the final tool. Figure  2  shows a flowchart that explains the different steps required to implement the crop spectral signatures database interactive tool (CSSIT). Information is displayed as graphs of spectral signatures, tables listing different wavelengths and reflectance values, crop photos at the time of data acquisition, and much more information.
To populate the database with data and metadata, the task of collecting field samples starts before the tool implementation, and it is based on user requirements (farmers, decision makers, and researchers) and must continue to do so in the future. Normally, after the requirement analysis and collection phase, the project leader and the development team can start to implement the conceptual design where one of the expected results is the entity relationships (ERs). Figure  3  shows the relationship between different entities that make up the database management system (DBMS). As an example, in the "metadata" table the item "NameofFile" refers to the name of the file which contains the spectral signature of the crop. Several metadata can be recorded such as the spectroradiometer's field of view (FOV), sky status, surveillance angles, time of data collection, and supplementary crop parameters.
In addition, important issues must be recorded such as file name and file size on the disk, format for recording radiance or reflectance, and whether any derivative is applied. Significant progress has been achieved and an initial interface has been created to be deployed on the Web in the future. The tool allows guest users for limited access. Special users are registered users who can have wider access to specific areas and features of the database. Finally, there is an administrator account that can modify content of the database and can add more collected data. Users with more privileges may have the right to backup and restore the database and maintain the tool. Figure  4a-d    As can be seen from the above figures when the application is launched, it requests authorization in order to allow the user to access its database and other existing functions and features.
After gaining access to the application (CSSIT), the guest user can select search item from the main interface window to launch the search dialog box. After searching for a specific crop and date of collection, one can either display the metadata of the crop or/and display the graph of the crop's spectral signature. If the user is authorized as an administrator then the tasks dialog box can be displayed to add, edit, or remove records. Finally, although the application does not allow complete access for guest or normal authorized users, in the future when it is deployed on the Web more features will be added such as a feedback and suggestions dialog box. A copy of the main application CSSIT under the name "setup. zip" is made available with this research paper in Supplementary Materials. The database is not included, but an update of the application may be provided to the journal in the near future.
Resampling of Spectral Signatures and Other Analysis The database can be used by experts to study crop characteristics and crop behavior during growth stage. Several spectral signatures for the same crop at different periods of time can be retrieved from the database using the tool to study the interaction of the crop with the incident electromagnetic waves (Figure  6 ). One can notice in the graph as the time progresses that as the wheat's leaves grow the reflectance percentage of near infrared increases. This is due to the increase in the area size of the leaves, and it means that the photosynthesis of the crop is higher. At the end, when the wheat crop approach harvesting time, the reflectance diminishes and the leaves turn a yellow color. A lot of important information can be obtained using the tool.
User Notes In the monitoring and mapping process of natural resources and agricultural products, a reliable geospatial technology supported by an up to date information database and metadata is needed. The advancement in technology represented by the existence of a reliable tool that can increase the efficiency of geospatial information extraction and reduce errors in decision making, can have a positive return for farmers. In this research, a new crop spectral signatures database interactive tool (CSSIT) is developed and in the enhancement phase. The objective of the tool is to help in the improvement process of crop mapping and in turn to ease the crop yield estimation task. It is shown in this paper that significant steps have been achieved toward completing the interactive database tool.