DCoM: A Deep Column Mapper for Semantic Data Type Detection
Authors: Subhadip Maji
Swapna sourav Rout
Sudeep Choudhary
Venue: ICLR 2022 Submitted (2022)
Keywords: Semantic Data Type Detection
Machine Learning
Natural Language Processing
Semantic Column Tagging
Sensitive Data Detection
Column Search

Abstract: Detection of semantic data types is a very crucial task in data science for automated data cleaning, schema matching, data discovery, semantic data type normalization and sensitive data identification. Existing methods include regular expression-based or dictionary lookup-based methods that are not robust to dirty as well unseen data and are limited to a very less number of semantic data types to predict. Existing Machine Learning methods extract a large number of engineered features from data and build logistic regression, random forest or feedforward neural network for this purpose. In this paper, we introduce DCoM, a collection of multi-input NLP-based deep neural networks to detect semantic data types where instead of extracting a large number of features from the data, we feed the raw values of columns (or instances) to the model as texts. We train DCoM on 686,765 data columns extracted from the VizNet corpus with 78 different semantic data types. DCoM outperforms other contemporary results with a quite significant margin on the same dataset achieving a support-weighted F1 score of 0.925. 

Under review as a conference paper at ICLR 2022
DCOM: A DEEP COLUMN MAPPER FOR SEMANTIC DATA TYPE DETECTION
Anonymous authors Paper under double-blind review
ABSTRACT
Detection of semantic data types is very crucial task in data science for automated data cleaning, schema matching, data discovery, semantic data type normalization and sensitive data identification. Existing methods include regular expression- based or dictionary lookup-base basud- methods that are not robust to dirty as well unseen data and are limited to very less number of semantic data types predict. Existing Machine Learning methods extract a large number of engineered features from data and build logistic regression, random forest or feedforward neural network for this purpose. In this paper, we introduce DCoM, a collection of multi-input NLPP-baeed deep neural networks to detect semantic data types where instead of extracting a large number of features from the data, we feed the raw values of columns (or instances) to the model as texts. We train DCoM on 686,76. 765 data columns extracted from the VizNet corpus with 78 different semantic data types. DCoM outperforms other contemporary results with a quite significant margin on the same dataset achieving support-weighted Fl score of 0.925.
INTRODUCTION
Robust data preprocessing pipeline enables organizations to make data-driven business decisions. Most unstructured and semi-structured data after initial pre-processing is available in tabular format for further processing. These tabular datasets have go through a process of data curation and quality check before consumption. Companies apply standard data quality checks/rules on business- critical columns to access and maintain the quality of data. Additionally, some of these datasets may contain sensitive information e.g. Protected Health Information (PHI), Personally Identifiable Information(PII), etc and need to be masked/de-identiffn Social Security Numbers, First Names, telephone numbers, etc. The first step in this data curation and quality check process column level semantic tagging/mapping. There have been many attempts by Ramnandan et al. (0155), Limaye al. (2010), Puranik (2012). Hulsebos al. (019). and Zhang et al. (2019) to automate this process. Traditionally. Semantic tagging is usually done using handcrafted rule-based systems e.g. tools from Google (209)), Microsoft (2019), etc. In some cases, column descriptions are also available. However, as the data grows exponentially. the cost associated with maintaining a rule-based system also increases. The semantic data type tagging is still largely manual where data stewards manually scan through databases and map columns of interest. Most ML-based approaches e.g. Zhang et al. (2019). Huusebbs al. (2019) and Pham al. (2016) use corresponding metadata, atomic datatypes, column description along with handcrafted features as input data for training the model.
i.e. identifying all columns which are
We herein, propose DCoM-Deep Column Mapper; a generic collection of Deep learning based Semantic mappers to map the columns to semantic data types. These models take the raw values of the columns (or instances) as inputs considering those as texts and build NLP- deep learning architectures to predict the semantic type of the columns (or instances). We also extracted 19 engineered features and used those as auxiliary features to DCoM models. We refrained from extracting a large number of features because we wanted to make the DCoM models learn those features along with some interesting high-level features on their own, for better semantic data type detection. As we are exploring NLP in this problem, we make the leverage of using advanced NI.P-based deep learning layers models such as Bi-LSTM, BERT, etc.

---

Under review as a conference paper at ICLR 2022
2 RELATED WORK
2019) Data Studio use some regular expression based patterns for dictionary looks ups of column headers and values to detect limited number of semantic data types. Venetis et al. (2011) build database of value-type mappings, then assign semantic types using a maximum likelihood estimator based on column values. [Syed et al. (2010) use column values and headers to build Wikitology query to map columns to semantic classes. Ramnandan et al. (2015) use heuristics to first separate numerical and textual types, then describe those types using the Golmoggrov-Smimma respectively. [Pham et al. (2016) use slightly more features, including the Mann-W Whitney test for numerical data and Jaccard similarity for textual data, to train logistic regression and random forest models. Goel et al. (2012) use conditional random fields to predict the semantic type of each value within column, then combine these predictions into a prediction for the whole column. Limaye| et al. 2010) use probabilistic graphical models to annotate values with entities, columns with types, and column pairs with relationships. Puranik (2012) proposes "specialist approach" combining the predictions of regular expressions, dictionaries, and machine learning models. More recently, Yan & He (2118) introduced system that, given a search keyword and set of positive examples, synthesizes type detection logic from open source GitHub repositories. [Hulsebos et al. (019 built multi-input deep neural network model for detecting 78 semantic data types, extracting 686, 765 data columns from VizNet corpus from Hu et al. (2019). They extracted a total of 1,58 1,588 features for each column to train the model, thus resulting a support- support-weighted F1 score of 0.89, exceeding that of machine learning baselines, dictionary and regular expression benchmarks, and the consensus of crowdsourced annotations. Zhang et al. (2019) introduced SATO, a hybrid machine learning model to automatically detect the semantic types of columns in tables, exploiting the signals from the context well as the column values. It combines a deep learning model trained on a large-scale table corpus with topic modelling and structured prediction to achieve support- weeihhed and macro average Fl scores of 0.925 and 0.735, respectively, exceeding the state-of- the-art performance by a significant margin. Recently Deng et al. (2020) used structure aware Transformer encoder capture semantics and knowledge in large-scal scale data. Wang et al. (2021) proposed Table Convolution Network and Suhara al. (202I) proposed a multi-task learning approach (called Doduo) which outperform the existing results in semantic data type detection. Iida et al. (202T) devised a simple pre-training objective (corrupt cell detection) that learns exclusively from tabular data and reaches the state-of-the art on a suite of table-based prediction tasks.
Microsoft 2019) Power BI and Google
(K-S) test and Term Prequenne-Inveerss Document Frequency (TF-IDF),
While in this paper we do not experiment with the context information of the column values, our work is mostly aligned to Hulsebos et al. (2019). Keeping that in mind, we are planning to research and implement the context of the column values in tables our DCoM models for our future works. The paper is organized as follows: Section describes the data used for DCoM models. In Section 4 we present the details of data preparation and architecture for DCoM models. Section 5 contains the training, evaluation and inference procedures of DCoM models while Section [a discussed about the results of extensive experiments. In Section)) 7 we talk about some known limitations from the data as well as application point of view and finally, in Section X we present the concluding remarks and some future directions.
3 DATA
We have used the dataset prepared by Hulsebos et al. (2019) and compared our model performance with them considering their results to be baselines. This dataset contains 686,765 instances with 78 unique classes (or semantic data types). It is divided into 60/20/20 vaining/valldaationeesinn splits. The instance count for classes varies from 584 (affiliate class) to 9088 (type class). The count distribution for classes is shown in Figure of Appendix [A] data sample of the dataset is also shown in 7 of Appendix
To provide a more clear picture of the
PROPOSED METHOD
Prior approaches (Venetis al. 2011 Limaye et al., 2010) to semantic type detection trained models, such as logistic regression, decision trees, feedforward neural network (Hulsebos et al. 2019), extracting various features from the data. We, on the other hand, treated the data like natural
2

---

Under review as a conference paper at ICLR 2022
language (text) and used the data itself as the input to the model. We used very small number of hand-engineered features unlike Hulsebos et al. (2019). Therefore, our DCoM models have two inputs, the values of the instances as text natural language and hand- engiineee features. We present two types of DCoM models based on the way we feed the text input to the model.
4.1 DCOM WITH SINGLE SEQUENCE INPUT
In this subsection, we diseuss how the values of each instance can be fed the model as single sequence input to the DCoM. Considering the scenario, we can not simply pass the in- puts separating values of each input with any separator token. This gives the model wrong information about the sequence of the data resulting in faulty training and poor performance on the unseen data. For example, if we consider the last example from Table 7 and cre- ate the input, Deletes the property <SEP> Lets you edit the value of the property <SEP> Script execution will be stopped for the model, the model gets to learn that the value Lets you edit the value of the property has relative po- sition between Deletes the property and Script execution will be stopped, which is quite wrong because the values in instance do not have any relative position between them.
To mitigate the problem introduce permutations from mathematics. With permutation, order items from the set of n; items. Here n, items are all the values of an instance i, and r E [1, n]. Doing so, the above instance can be broken down into multiple subsets. A sample of the subset is shown in Table If we feed all the new instances of the subset to the model, it does not learn any relative positional information of Lets you edit the value of the property with respect to other values in that instance unlike earlier. Therefore, for the value Deletes the property. the model only learns the relative positions of the tokens e.g. Deletes, the, property, etc in value, but not the relative position of the values in an instance. This helps the model getting the actual information present in the data for accurate prediction on the output. This permutation method also helps in augmenting new instances which help in training the data-hungry deep learning models with enriched data. It is not feasible to generate all the possible permutations before the training because of the huge number of subsets. Instead, during training, we sample between I and n for each of the instances and generate one permutation for each instance. More than one instance can be generated, but training the model for multiple epochs will result same for both cases.
dass dosilption
Dalutes the propeacy
late you edit the va ie of the proparty Serigt exeeation will stopped
deccpttion
docepptionn desatiptinn
Peletis paoperily cages Leta you edi. Daletes the propecty <SEP> tats edit the propetty the propecty <SEP> Script execution willl anopped Deletas the propestly <SEPs Scripe execution will be stoppec <EEP> tate you ecit the valus of the propezty Lets you edit the value of the propercy casPs beletes the property <osPs Script ion will be scopped
value the property
stopped
dectppiion
deserption
desclption
Table 1: Permutations on the values of a single instance from class description.
4.2 DCOM WITH MULTIPLE SEQUENCES INPUTS
Input preparation with this method is straightforward compared to the earlier method. In this method, we also use permutations to generate new instances, but the value of is fixed during training and is used as a hyper-parameter, where r E [1, oo). Once we decide the value of r, number of inputs used as text inputs to the model. Aggregation of embedding vectors of the inputs are performed once they are generated using the shared embedding weights. If r > n. where n is the number of values of an instance, then this scenario can be handled in two ways. The first way is to pad r n inputs and aggregate the embedding vectors only for the non-padded inputs. Another way is while generating new instances use permutation with replacement to always sample r values out of values. Therefore, padding is not required for the latter method. We tried both approaches and did not observe any significant difference in the result.
We used 19 out of 27 global statistical features from Hulsebos al. (019)) as our engineered features for the DCoM models because we are only considering the unique values of columns. These features normalized before feeding to the model. The complete list of these features is shown in Table [5] Once the text and engincered inputs are prepared, we attach STMM/ransformer/BERR layers to the text input. The output of the earlier layers is aggregated with the engineered inputs. We use
3

---

Under review as a conference paper at ICLR 2022
some feed-fore woed-fo layer after that. Finally, probability of each of the classes as output. This is our generalized architecture design of DCoM models. Extensive hyper-parameter tuning is performed to finalize the number of layers, number of units in a layer and many other hyper-parameter in the model. This topic is discussed in detail in section[5] [ To name the DCoM models, the type of the text input and the name of the deep learning layers are used as suffixes with the name DCOM, e.g. DCoM model with single instance input and LSTM layers are named as DCOM- SIMM--Singe Single -LLMM. The architecture design of DCoM models for both single sequence and multiple sequences inputs are shown in Figure
one softmax layer with 78 units to get the
TOAR The lundey
Mat
Apmud
JPM
Print
Figure Architecture diagram of (a) DCoM- and (b) (CoM-MMlti i models.
5 TRAINING, EVALUATION AND INFERENCE
We trained our model on the train dataset, validated it on the validation dataset and finally reported our results on the test dataset. As class imbalance is present, like Hulsebos et al. (2019), we also evaluated our model performance using the average F1-score, weighted by the number of columns per class in the test set (i.e., the support).
The DCoM models are trained in Tensorflow (Abadi et al. 2015) and the hyperparameters are tuned using Keras-tunel (O" (O'Malley et al. 2019). For inputs of DCoM models, we tried permutations with replacement as well as without replacement, but we did not observe any significant difference in outputs. The value of r for which we observed best performance for DCoM-Mu 1 models is 45. For tokenizing text inputs, we experimented with searratter-bssed word-based and BERT Wordpiece (Wu) ct al. 2016) tokenizers, and we found out BERT Wordpiece tokenizer stood out to be working better with respect to the other tokenizers because of the obvious reasons stated the paper (Wu et al.. 2016). We compared our result with and without pre-trained embedding weights. It is observed that, though in the final output there is no considerable difference, training with pre-trained embedding weights take approximately 40% less amount of time to converge. We experimented with the small, base and large variations of several BERT architectures and finalized Disti i BEBT- (Sanh) ctt al., 2019) and plectra-small Bi-directional LSTM layers in all the DCoM-LSTM variants. For Dropout layers we used 0.3 as our dropout rate. We experimented mean, sum, concatenat i on and we thigh sum functions for aggregation, but there was not any significant difference in outputs based on these variations. We used Adam (Kingma & Ba 2014) optimizer with initial learning rate 10-4, Along with this, we implement learning rate reducer with factor of 0.5, which reduces the learning rate by 50% if the model performance does not improve on the validation dataset after consecutive 5 epochs. Assigning class weights does not have much effect on the test results. As this is multi-class (78 classes) classification problem, the DCoM models are trained with categorica aacal-eres well--rosseenn -crossenzropy loss and validated with accuracy and average F1 score metric.
(Clark al., 2020) based on their performances. We used
4

---

Under review as a conference paper at ICLR 2022
We tried two approaches for inference the test dataset. In the first approach, performed single time inference on each the instances of the test dataset. While doing so, we set r to be the total number of values (n,) for each instance for DCoM-Single models to perform inference. For DCOM-Mult.. as r values if prefixed, some values will be truncated for inputs with total values The other approach is generate k (where, k > 1) instances with permutation, sampling values k times between 1 and n,, where n& is the total number of values for instance i. Therefore, we get & class predictions for each of the instances and finally with majority voting we pick the prediction class for each instance. We used & value to be 10 in our case and it is observed that for both DCOM-S Single e and DCoM-Multi, we observed 0.2 0.5% improvement in the test average Fl score, but this improvement comes with the price of increased inference time by approximately k times.
RESULTS
We compared DCoM against Sherlock and other types of models shown by Hulsebos et (2019) along with SATO Zhang et al. 2019), TURL (Deng et al. 2020). TABBIE (lida et al. 2021). DODUO (Suhara et al. 2021) and TCN (Wang al. al.] assuming those to be our baseline models. All the models are compared on the dataset used by Hulsebos et al. (019) in Sherlock paper. Table presents the comparison of results on the test dataset for each of the models. Columns Engineered Features and k are specific to DCoM models. Engineered Features says whether the 19 engineered features are used with the text inputs while training the model. k column tells the number of times inference is performed on single test instance. It is discussed in detail in section Runt i me column is the average time in seconds needed to extract features and generate a prediction for single sample, and S ze column reports the space required by the trained models in MB. From the table, it is seen that several DCoM models outperform all the bascline models in both Fl score and inference run time with significant margins.
Hulsebos et al. (2019) extracted various features e.g. global statistics, character-level distributions, word embeddings, paragraph vectors from the data and used those features fit a feedforward neural network model. On the other hand, we treated the data as texts and feed those to more advanced NLP. based models. This allows the DCoM models extract and learn more useful features that Huulsebss et al. (019)) were unable to extract using hand-engineering, Along with this, hand-engineering takes a considerable amount of time to calculate the features which increase the inference time of Sherlock (Hulsebos et al. 2019) by 3 to 20 times with respect to DCoM models.
Method
Engineered Features
F1 Score
Runtime (s)
Size (MB)
ICoM--1nglee-18it
0.895
0.019 0.01
BCoM-Singlee-IS1
Yes
10
0.898
0.152 0.01
112.1
DCOM-Single-ISSW
No
1
0.871
0.018 : 0.01
97.8
DCoM-Singg1e-1117
0.877
0,141 0.01
97.8
RCoM-MMu1ti-1.87M
Yes
1 0.878
0.046 0.01
4.7
DCoM-MulLi-LsTH
Yes
10
0.881
0,416 0.01
4.7
DCoM-Multi11STT
I 0.869
0.044 0.01
4.6
No
0.871
0,401 0
4.6
#cof-SingleeDDisti1BRRR
Yes
I 0.922
0.162 0.01
268.2
DCoX-BBingLe-DDsttLLBBR
Yes
0.925
1,552 0.01
268.2
ScoM--Sinngle-Disii11BBR
No
0.901
0.158
202.3
No
10
0.904
1.492 0.01
202.3
IKoM--Single-91lsttr
Yes
0.907
0.093 0.01
53.1
BCoK--Single-Blesttaa
Yes
0.909
0.894
53.1
DCo#-SingLe-Bleetze
No
0,890
0.092 0.01
45.7
RcoM-Siinqle-1lecct
No
10
0.892
0.887 0.01
45.7
SATO (Zhang et 2019
0.891
0.51 0.01
25.7
TABBIF: (Ida
X02
0.897
0.058 0.01
613.4
DODUO Subbraa et al. 2021
0.904
0.020 0.01
155.8
TCN (Wang et al 202
0.897
0.017 0.01
108.7
TURI Deng
2020
0.894
0.017 0.01
110.2
Sherlock (Hukeebo et al 2019
0.890
0.42
6.2
Decision tree Hulssbos al 2019
0.760
0.26
59.1
Random Forest Forest [Hulscbos
0.840
0.26 0.01
760.4
Dictionary (Hul at 2019
0.160
0.01 + 0.03
0.5
Regular expressool (Elissc
2019
0.040
0.01 + 0.03
0.01
Consensus (Hulsebos
0.320
33.74 0.86
Table 2: Support-weighted F1 score, runtime at prediction, and size of DCoM and other benchmarks
5

---

Under review as a conference paper at ICLR 2022
Type
FI Score
Precision
Recall
Support
ISBN
0.988
0.999
0.993
1430
Grades
0.988
0.995
0.992
1765
Top 5 Types
Birth Date
0.987
0.985
0.986
479
Jockey
0.983
0.988
0.986
2817
Industry
0.984
0.984
0.984
2958
Person
0.813
0.596
0.688
579
Rank
0.586
0.764
0.663
2978
Bottom 5 Types
Director
0.702
0.549
0,616
225
Sales
0.667
0.435
0.526
322
Ranking
0.695
0.348
0.464
439
Table 3: Top and bottom five types by Fl score the test dataset for DCoM-Sing Singlennggie.ie B model variant.
Examples
True Type
Predicted Type
1, 4,5,4,32
Day
Rank
Low Precision
7,
Region
Rank
1, 2,
Position
Rank
2,, 36
Ranking
Rank
Low Recall
0, 2,4
Ranking
Plays
2, 3, 4, 6, 7,
Ranking
Rank
Table 4:
Examples of low precision and low recall types on the test dataset for UCoM-Single-DDstiiBBB. model variant.
6.1 PERFORMANCE FOR INDIVIDUAL TYPES
Following Hulsebos et al. 2019). we also prepared Table which displays the top and bot- tom five types, as measured by the Fl score achieved by the best performing DCoM model, DCOM-S meinngle-Die e- Ditttiiiii BERT for single inference (k = 1) that type. High performing classes such as grades, industry, ISBN, etc. contain a finite set of valid values which help DCoM models to extract distinctive features with respect to the other classes. understand types for which DCOM- Single Soiggle-Distio Distiiissts performs poorly, include incorrectly predicted examples for the lowest precision type (Rank) and the lowest recall type (Ranking) in Table 4 From the table, is observed that purely numerical values values appearing in multiple classes, cause challenge for the DCoM models to correctly classify the semantic type of the data. This issue is discussed in detail in section
6.2 FEATURE IMPORTANCE
To calculate the feature importance of 19 engineered features used in trained WCoM-Single-DistillBB
model, we extracted the learned feature weight matrix, W from the dense layer used after the engineered features input, where W I11x Here D is the number of units used the dense layer. contains the weights/contributton of each of the 19 features each of the D units, thus forming 19 x D array, which contains both positive and negative values based on the direction of contribution. We take the absolute value of all the elements of W, are only interested in the amount of contribution of each of the engineered features, not the direction. After that, we take the mean across the 19 features that result in a 19-dimensional array. We normalized the array by dividing the maximum value of the array by each of the elements. Table 5 enlists the importance of the engineered features in decreasing order. The feature importance score of the engineered features changes with the different variants of DCoM models but the rank of the top 10 features remain intact.
7 KNOWN LIMITATIONS
The major limitation of the process is with the pre-processed data prepared by Hulsebos et al. (2019) used to train DCoM models. The same values of instances present in multiple classes, thus

---

Under review as a conference paper at ICLR 2022
Rank
Aux Features
Score
Std of # of Numeric Characters in Cells
1.00
Std of of Alphabetic Characters Cells
0.63
3
Entropy
0.63
4
Std of # of Special Characters in Cells
0.62
5
Std of of Words in Cells
0.53
6
Mcan # Words in Cells
0.50
Mean of Numeric Characters in Cells
0.48
8
Minimum Value Length
0.45
Kurtosis the Length of Values
0.41
10
Mean # Special Characters in Cells Number of Values
0.39
0.34
12
Fraction of Cells with Alphabetical Characters
0.33
13
Fraction of Cells with Numeric Characters
0.31
14
Sum of the Length of Values
0.31
15
Maximum Value Length
0.30
16
Skewness of the Length of Values
0.28
17
Mean # Alphabetic Characters in Cells
0.28
18
Median Length of Values
0.27
19
Mode Length of Values
0.20
Table 5: Feature importance of 19 engineered features used in DCoM-Singg Single-Distiiiieeeeeeee
model.
Values
Class
F, M
gender
M
sex
gender
M.
sex
1, 3, 4, 7,8,
10,
11, 12, 13, 14
ranking
2, 3, 4, 6, 8,
10,
11,
12, 13, 14
position
1. 3, 6, 7, 9.
10, 11, 12. 13, 14
rank
Fresno State, Oregon, UCLA, South Carolina
team
Michigan, Indiana, Wisconsin, Purdue
team name
Table 6: A sample example of class overlapping of the data used to build up models
resulting in strong class overlapping among classes. It makes the models confusing during training to learn distinctive features for proper classification. Therefore, for some classes in the test dataset, the models perform very poorly. Table presents sample of examples of class overlapping in the data prepared by Hulsebos et al. (2019). It is to be noted that the proportion of this class overlap considerably large numbers with respect the total number of instances. Therefore, it affects the training as well as the model performance on the test dataset by significant margin. Besides the class overlap, faulty or wrong values are present some classes.
From the application point of view of semantic data type detection models, identifying and preparing good dataset for training is very challenging as well as time-consuming. Kon-standardized column names are major challenge large organizations face in doing semantic detection. For example, social security numbers can be called as ssn, ssn_id, soc_ coc_sec_bnz etc. Non- standardized data columns also pose major challenge in information ambiguity. For example, Gender column can have values Male, Female, unknown, or 0, 1, 2. Some organization have mixed attributes which poses major challenge in data security. For example. of PII data like Social security Number in Customer ID columns. Some other challenges like corrupt/missing metadata also exist.
7

---

Under review as a conference paper at ICLR 2022
8 CONCLUSION
DCoM presents a novel permutation-bant based method by which the instances can be fed to the deep learning models directly as natural language. This takes the leverage of using more advanced NLP- based layers/models unlike feedforward neural networks in Sherlock Hulsebos et al. 2019). The permutation based method also helps in generating large number of new instances from the existing ones, helping DCoM models to boost their performance effectively, thus outperforming Sherlock (Hulsebos et al. 2019) and other recent works by quite significant margin in both Fl scores as well as inference time. We also present ensembling approach during inference time which improves the test average
score by 0.2 0.5%.
As mentioned earlier, the next step of it to introduce the context of the column values to DCoM models while predicting the semantic types in relational tables.
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving. Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Mike Schuster, Jonathon Shlens, Benoit Steiner, llya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL https: /www www.ttenssoooooo org/ Software available from tensorflow.org.
Olah,
Kevin Clark, Minh- Than- Luong, Quoc V. Le, and Christopher D. Manning. ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators. arXiv e-prints, art. arXiv:2003.00555, March 2020.
Xiang Deng, Huan Sun, Alyssa Lees, You Wu, and Cong Yu. TURL: Table Understanding through Representation Learning. arXiv e-prints, art. arXiv:2006. 14806, June 2020.
Aman Goel, Craig A. Knoblock, and Kristina Lerman. K.: Exploiting structure within data for
accurate labeling using conditional random fields. In In: Proceedings of the 14th International Conference on Artificial Intelligence (ICAI, 2012.
Google. Google data studio, 2019, URL htppttt//aaattttiiooooooo
Kevin Hu, Neil Gaikwad, Michiel Bakker, Madelon Hulsebos, Emanuel Zgraggen, César Hidalgo, Tim Kraska, Guoliang Li, Arvind Satyanarayan, and Çağatay Demiralp. Viznet: Towards a large- scale visualization learning and benchmarking repository. In Proceedings of the 2019 Conference on Human Factors in Computing Systems (CHI). ACM, 2019.
Madelon Hulsebos, Kevin Hu, Michiel Bakker, Emanuel Zgraggen, Arvind Satyanarayan, Tim Kraska, Çagatay Demiralp. and César Hidalgo. Sherlock: deep learning approach to semantic data type detection. In Proceedings of the 25th ACM SIGKDD International Conference Knowledge Discovery &#38; Data Mining. ACM, 2019.
Hiroshi lida, Dung Thai, Varun Manjunatha, and Mohit lyyer. TABBIE: Pretrained Representations of Tabular Data. arXiv e-prints, art. arXiv:2105.02584 May 2021.
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. arXiv e-prints, art. arXiv: 1412.6980.. December 2014.
Girija Limaye, Sunita Sarawagi, and Soumen Chakrabarti. Annotating and searching web tables using entities, types and relationships. Proc. VLDB Endow. 3(1-2): September 2010. ISSN 2050-8097. doi: 10. 11..4477//1208411999999
URL httpt:tttt/.....tt /t0
1920841. 1111.19210 1921005
Microsoft. Power bi, interactive data visualization bi, 2019. URL https:/ httpp:::///oweeb pwwerbi mi miccrooft.com com
8

---

Under review as a conference paper at ICLR 2022
Tom O' O'Malley, Elie Bursztein, James Long, François Chollet, Haifeng Jin, Luca Invernizzi, al. Keras Tuner. htttps://githbb ttantaaaao.cocceeaateeeeaaaaa/ea
2019.
Minh Pham, Suresh Alse, Craig Knoblock, and Pedro Szekely. Semantic labeling: A domain-
independent approach. pp. 446-462, 10 2016. ISBN 978-3-319-46522-7.
doi: 10.1007/
978-3-319-44522-44222
N. Puranik. A specialist approach for the classification of column data. 2012.
S.K. Ramnandan, Amol Mittal, Craig A. Knoblock, and Pedro Szekely. Assigning semantic labels to data sources. In Fabien Gandon, Marta Sabou, Harald Sack, Claudia A'mato Philippe Cudré- Mauroux, and Antoine Zimmermann (eds.), The Semantic Web. Latest Advances and New Domains, pp. 403 417, Cham, 2015. Springer International Publishing. ISBN 978-3-319-18818-.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv e-prints, art. arXiv:1910.01 October 2019.
Yoshihiko Suhara, Jinfeng Li, Yuliang Li, Dan Zhang, Çağatay Demiralp, Chen Chen, and Wang- Chiew Tan. Annotating Columns with Pre-trained Language Models. arXiv e-prints, art. arXiv:2104.01785, April 2021.
Zareen Syed, Timothy W. Finin, Varish Mulwad, and A. Joshi. Exploiting a web semantic data for interpreting tables. 2010.
Petros
Venetis, Alon Halevy, Jayant Madhavan, Marius Paşca, Warren Shen, Fei Wu, Gengxin Miao, and Chung Wu. Recovering semantics of tables on the web. Proc. VLDB Endow, (9):528-a))) 538, June 2011. ISSN 2150-8097, doi: 10. 10.14778/20022388200022 2002999,, URL https: 7/doi oro.:.org/00... http://....17702229922222022
Daheng Wang, Prashant Shiralkar, Colin Lockard, Binxuan Huang, Xin Luna Dong, and Meng Jiang. TCN: Table Convolutional Network for Web Table Interpretation. arXiv e-prints, art. urXiv22102.09460 February 2021.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang. Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. arXiv e-prints, art, arviv: (609.08144 144, September 2016.
Cong Yan and Yeye He. Synthesizing iypeddetection logic for rich semantic data types using open- source code. In Proceedings of the 2018 International Conference Management of Data, SIGMOD '18, pp. 35-50, New York, NY, USA, 2018. Association for Computing Machinery, ISBN 97711773117700 doi: 00.1145/1187713331966 F183713.31951
Dan Zhang, Yoshihiko Suhara, Jinfeng Li, Madelon Hulsebos, Çağatay Demiralp, and Wang-Chiew Chiew Tan. Sato: Contextual Semantic Type Detection in Tables, arXiv e-prints, arXiv: 1,0631,, November 2019.

---

Under review as a conference paper at ICLR 2022
A DATASET DETAILS
10000
9000
7000
of Instences
5400
4000
2000
Loperggo
Mipestoo
Class (Total 78 in number)
Figure 2: Count of instances for each of the 78 classes in the dataset Hulsebos et (011
Instance 2, 4, 6, 7,8
Class
day
4:59, 2:44, 2:04, 2:05, 1:14
duration
Education, Poverty, Unemployment, Employment
category
c, end-code, code name, label name
command
years, 22 years, -, years, years 1.2
age
position
CA, AL, Warwickshire]
State
Deletes the property, Lets you edit the value of the property. Script execution will stopped
description
Table 7: A sample of the dataset.
10


Reviews:
[{"review_id": "k75nYXKXCGP", "reviewer": "Reviewer_MWJi", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strength\n\n- (S1) The paper combines learned representation and manually crafted features for further improvements on the semantic detection task.\n\n\n### Weaknesses\n\n- (W1) The novelty and technical significance are limited.\n- (W2) Missing important references and comparison against the state-of-the-art methods.\n- (W3) DCoM-Multi does not show improvements over DCoM-Single.\n\n### Major comments\n\n(W1)\n\nUsing raw table values for semantic type detection is a commonly used approach and several models have been already developed [1-4]. The models in [1-3] also incorporate table structures (i.e., row/column) into the model, which is not considered in DCoM.\n\n- [1] Xiang Deng, Huan Sun, Alyssa Lees, You Wu, Cong Yu, TURL: Table Understanding through Representation Learning, VLDB 2021 (https://arxiv.org/abs/2006.14806)\n- [2] Hiroshi Iida, Dung Thai, Varun Manjunatha, Mohit Iyyer, TABBIE: Pretrained Representations of Tabular Data, NAACL 2021 (https://arxiv.org/abs/2105.02584)\n- [3] Daheng Wang, Prashant Shiralkar, Colin Lockard, Binxuan Huang, Xin Luna Dong, Meng Jiang, TCN: Table Convolutional Network for Web Table Interpretation, WebConf 2021. (https://arxiv.org/abs/2102.09460)\n- [4] Yoshihiko Suhara, Jinfeng Li, Yuliang Li, Dan Zhang, Çağatay Demiralp, Chen Chen, Wang-Chiew Tan, Annotating Columns with Pre-trained Language Models, arXiv April 2021 (https://arxiv.org/abs/2104.01785)\n\nFurthermore, all the models and Sato [Zhang et al. (2019)] are multi-column models that consider all columns in the same table to detect the semantic types in a joint manner. As discussed in [Zhang et al. (2019)], the table context is essential to semantic type detection. \n\n\n(W2)\n\nAs commented above, the paper should cite and compare with [1-4] in addition to Sato, which is cited but not compared in the paper. \n\n\n(W3) \n\nAccording to Table 3, DCoM-Multi does not show improvements over DCoM-Single. \n\n\n### Minor comments\n\n- Why did the author(s) use the 19 features out of 27 Sherlock features?\n- Do the feature importance scores for different models (e.g., DistilBERT) follow the same or a different trend against the feature importance scores reported in Table 6?\n- Typo: we -> We (p.5 2nd paragraph)\n", "summary_of_the_paper": "This paper develops a semantic type detection model DCoM that takes column values as text in addition to auxiliary features extracted by the exiting semantic type detection method Sherlock. The paper proposes two versions: (1) Single, which takes a single sequence, and (2) Multi, which takes multiple sequences using permutation. Experimental results on the VizNet corpus show that DCoM models with DistilBERT/ELECTRA outperform Sherlock and other non-Deep Learning models. ", "summary_of_the_review": "The paper has critical issues with respect to novelty and technical significance. The paper should cite the missing papers and compare with those models (and Sato.) Thus, I wouldn’t recommend publishing the paper in its current form. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "ZhtMCE6eYLm", "reviewer": "Reviewer_S7C6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- The task is interesting/important\n- Good performance is reported\n\nCons:\n- While the the context information of the column values is of interest, modeling this information is deferred to future work.\n- Classifying the semantic type based only on the string value is too simplistic. One of the main challenges of this task in practice is to resolve ambiguities, where the authors do not address this aspect altogether. \n- Assuming that the classifier/s may just memorize values per category, the authors should check/report the overlap between train and test.\n- Otherwise, it would be interesting to train and test on different datasets, to assess learning generalization.\n- I'm not sure that the dataset contains enough diverse examples for learning 70+ categories.\n\nSome typos:\n- many attempts by Microsoft (2019): cannot cite like this.. (repeats several times)\n- We refrained ourselves extracting --> refrained from\n- The output of the same is aggregated --> ??\n- Figure 2 uses a too small font.\n", "summary_of_the_paper": "The goal of the work is to associate semantic data types to table attributes. This task is of importance to data cleaning, schema matching,  etc. The authors claim that existing methods use regular expressions or dictionary lookups, and proposed a learning approach to the problem.\nThey feed raw values (strings) to neural classifiers, and train their models on 686,765 data columns extracted from the VizNet corpus with 78 different semantic data types. The textual values processed by LSTM/Transformer/BERT layers, where in addition the authors use 19 tatistical features from Hulsebos et al. (2019).  Overall, they achieve F1 score of 0.925.\n\nAnalysis showed that high performance is obtained for classes that contain a finite set of valid values, e.g. grades or industry. \nOn the other hand, numerical values or in general values that appear in multiple classes, are more challenging to classify. \n", "summary_of_the_review": "The approach is overly simplistic, as it does not account for ambiguities. Methodologically, I'm not convinced that the model generalizes, as there may be significant overlap between the train and test examples.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DOjIs5_ryOU", "reviewer": "Reviewer_S4in", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Main Review\n\nThe paper studies a very practical problem called semantic type detection. The problem is fundamental to tabular data that consists of columns and headers. Given the widespread adoption of tabular data in database systems, the paper has the potential to benefit extensive applications. The paper presents a new method DCoM for the problem. DCoM avoids the feature engineering step, which is common in existing methods. Also, DCoM achieves satisfactory performance that hits a support-weighted F1 as high as 0.925. However, the paper can be more convincing if several aspects can be taken into consideration. Details are as the following:  \n\n1.\tPretrained models should be included as baselines. The proposed DCoM adopts a pre-trained model in the design. For example, DCoM-Single-DistillBERT leverages the pre-trained model DistillBERT that comes from HuggingFace Transformers library. The library contains other pre-trained models, including BERT, RoBERTa, and ALBERT. The interface of using these models is the same as DistillBERT. It will be interesting to see whether different pre-trained model leads to different performance of DCoM, and more importantly, whether these models standalone can solve the semantic type detection problem adequately. The paper will be more convincing if it can include the support-weighted F1 scores of one or more of these pre-trained models as baselines. For experiments, the semantic type detection problem can be naturally formulated as sequence prediction, which is the strength of pre-trained models.\n\n2.\tAblation study is favorable. DCoM presents impressive performance when compared with existing methods. However, it remains unclear why DCoM can achieve superior performance. Is it because DCoM uses higher-dimensional word vectors or DCoM uses larger neural network? The paper can be more insightful if it conducts some ablation study to find out why DCoM is superior.\n\n\n3.\tReal applications can be discussed further. Semantic type detection seems a fundamental problem in tabular data, and an immediate application is header prediction. However, header prediction may not be common because the headers and columns in a table are usually serialized at the same time. Semantic type detection can be more useful if it can affect many downstream tasks (e.g. column matching and schema matching). Some discussions on applications will be helpful to make DCoM adopted extensively in the industry.\n", "summary_of_the_paper": "This paper studies the problem of semantic type detection that takes a column of texts then predicts a header name for the column. Existing methods mostly adopt regular expressions or dictionary-lookup or feature engineering. The paper introduces a new method, called DCoM that is different from existing methods. DCoM adopts deep neural networks to process raw text tokens directly instead of relying on feature engineering. The paper conducts extensive experiments and shows that DCoM outperforms a number of contemporary methods on VizNet dataset, which has 78 types. ", "summary_of_the_review": "Summary of the review\nThe paper studies a novel problem called semantic type detection, which has potential applications over tabular data. However, the paper should include pre-trained models as baselines to show the proposed DCoM outperforms the state-of-the-art methods by a significant margin. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies semantic type detection.
 The problem is of practical significance to  i  tabular data.
 However, in its current form, there are concerns about  the scope of novelty and technical significance.
Decision: Reject