Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: null
review_content: This kind of dataset is a valuable contribution, and authors' thoughtful approach to assembling and constructing it is admirable. There are no perfect choices when constructing or employing a training data set, and the sampling methods and de-identification choices (particularly the exclusion of small organizations) used here are thoughtful and well-designed to minimize harm. 
review_title: brief, 11th-hour ethics review

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper introduces The Labour Inspection Checklists Dataset consists of 63634 instances where each instance is an inspection conducted by the Norwegian Labour Inspection Authority. 575 features are provided for each organization that is inspected and potential target variables are checklists (list of items that should be inspected for the organization) and non-compliance. The authors experiment with several machine learning methods on their task, but the performance for all of them is very poor. 

I appreciate the authors' response, and I have updated my score. However, I still believe that the extremely poor performance of ML models on this dataset is a red flag. The dataset in the current format is probably not informative enough, which can even discourage researchers from working on this area as the trained models don't generalize. I would encourage the authors to include other sources of data if possible to mitigate this issue.
strengths: 1. The authors are releasing a dataset on a relatively unexplored task; however, it could be helpful for better inspection by agencies.
2. The dataset consists of 63634 inspections which is to some extent large.
3. Several experiments are conducted among two tasks, predicting checklists and non-compliance with different machine learning models.
weaknesses: 1. The whole data is only coming from a single authority (NLIA). Therefore it might not be directly useful for other parts of the world.
2. The performance of the trained machine learning models is extremely poor. That probably could be because the data is not informative enough for the defined task. Poor performance sometimes means that the data is not informative enough, which I think is the case here.
3. The authors do not elaborate on features that are provided for organizations and why they believe they are informative enough. Also, for the checklist, I think there is no single ground truth for it. The fact that a set of checklists are selected during the inspection by the inspector does not mean that the chosen set is optimal, making predicting the list even harder.

correctness: The claims seem correct to me.
clarity: The paper is not well written in my opinion. I think the authors can do a better job presenting sample input (features) and outputs (checklist) in the paper for their defined task.
relation to prior work: The authors mention previous datasets in this domain such as "Machine learning for environmental monitoring" by Hino et. al.

additional feedback: I think in the current format the data is not informative enough for the task that the authors have in mind. Changing the input features or the desired output can help solve this issue.
review_title: The datasets is not informative enough

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper introduces a new labor inspection checklists dataset (LICD) for selecting relevant checklists to address working environment violations in organizations, which is very important for protecting labor rights. LICD consists of 63634 instances with past inspections and contains 575 features and 2 possible target variables: Non-compliance and checklists. 
Based on the target variables, the dataset could potentially be used to select a relevant checklist to (1) survey a given target organization as well as to (2) classify whether non-compliance is found at an inspection, for a given organization and checklist. It could be used by inspectors to select high-risk organizations for inspections. 
The authors used feature selection in the experiments with Anova F, χ 2, coefficient importance from model training, Mutual information, sequential selection (forward search), and recursive elimination. The ML methods used for the experiments were Decision tree (DT), Logistic regression (LR), Naive Bayes’ Classifier (NBC), K-Nearest-Neighbor (k-NN), AdaBoost, Gradient Boost, and Multi-layered Perceptron (MLP). GridSearchCV for hyperparameter tuning for k-NN, AdaBoost, GradientBoost, and MLP. Each ML method was evaluated on 8 different feature set sizes that are selected via feature selection. There are only minor differences between most of the methods in the test. Sequential Selection, Anova F and Mutual Information have the best-recorded accuracy scores.

Future work using this dataset could, for example, involve a multi-objective optimization problem to select the most relevant inspection checklist that maximizes the number of violations found in a given organization. 
strengths: This new and unique dataset is addressing a very important labor safety issue, promoting a safe working environment for all workers and motivating further research on this subject.
The authors conducted initial features evaluation and basic data visualization to highlight how some of the key features of the dataset are distributed. 
The high number of relevant checklists used within a single industry code points to the fact that there is significant diversity in the health, environmental, and safety risks for organizations, even within the same industry.

Good job on being specific about the equipment used: “A Dell Precision 5560 laptop with Intel i9 11950h at 5Ghz, 64GB RAM at 3200Mhz, Nvidia Quadro RTX A2000, and Windows 10 are used for the experiments.”

The full dataset was imbalanced in terms of non-compliance. To address this issue, the authors created a balanced set for ML models.
weaknesses: 
Several questions were not fully addressed in the manuscript:

*How was the data balancing created? Which samples were selected? 
*Were there any work done for the analysis of the similarities between the users and industries? 
*Do the industry code numbers also maintain the similarity information between the industries? Are industries that have numbers that are closer also perform more similar business rather than distant codes? What could be the reason for the spike gaps between codes in Figure 2a for example, between 50 and 60, which correspond to most of the building and road construction industries? Could it be better to treat them as categorical features rather than numeric?

*Sequential Selection was unable to complete within two hours for any of the larger feature sets. Would running the selection longer help?

The results for the trained models on the unbalanced LICD have room for better performance and highlight the necessity for running the experiment on a balanced dataset or more complex models. However, the results on the balanced data improved also leave room for improvement. Details about the GridSearchCV hyperparameter tuning were not specified in the text. Thus, better results could potentially be achieved using a broader or better hyperparameters tuning mechanism in future work.

The variance for the cross-validation runs is not shown in the manuscript. Could you please add it?

correctness: 
The main claim about the importance of the dataset seems to be accurate. The authors provided a broad set of models that could be used as well as feature selection mechanisms.

It was mentioned that each configuration is evaluated using 5-fold cross-validation with randomly selected training evaluation sets from LICD. The performance is measured in terms of balanced accuracy, accuracy, precision, and recall scores using the available methods in the Scikit-learn library. The results for each method are recorded by calculating the average of the scores reported from the 8 feature selection configurations. However, the variance was not found in the main text.

The authors were upfront and clarified: “All in all, none of the methods above perform well in terms of classification performance on either LICD or LICD/B. For this domain, a high precision score is important because it means that a higher percentage of non-compliance is found on average per inspection. A high recall is also desirable, but less important as long as there are a sufficient number of predicted positives. A decent accuracy is also important, in order to ensure that each method has decent classification performance.”

clarity: The paper is well written, but some details are missing.
relation to prior work: Prior work was described in relation to the inspection domains, existing data, and ML models.

additional feedback: 
Thank you very much for working on such an important topic and creating a new dataset and bringing attention to the safety of work environments.
review_title: A very important topic and dataset for protecting labor rights and safe working environments. Additional information is needed.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This work seeks to offer a new dataset on over 60,000 workplace inspections as well as completed checklists related to labor law compliance in Norway between 2012 and the beginning over 2019. This dataset is intended to help identify patterns of health, safety, and environmental violations using ML techniques as well as help determine which are the appropriate checklists to use in a given context. The authors note that the sample experiments that they show using this dataset unfortunately do not exhibit a high performance with respect to predicting violations but are optimistic that novel methods using this data may be able to outperform their benchmark experiments. 
strengths: The manuscript illustrates that one contribution of this dataset it “expands on previous SDG related NeurIPS datasets and benchmarks (such as SustainBench),” since none of those datasets cover SDG 8. This is a notoriously difficult arena to get good data on (especially financial information of regulated entities), and I laud their efforts to make this information public. 

weaknesses: This dataset is focused in on one country only. This is still an advance, as I appreciate this data is difficult to obtain, and regulations vary from place to place that will characterize what is in “violation” or not differently. However, one major difference between this, and, for example SustainBench that they mention in related work, is that the latter seeks to at least keep track of more than one country. 

While they contextualize this dataset within other related work, it’s not clear to me that all of the citations refer to datasets (some appear to refer to manuscripts that illustrate analysis, others to datasets that may be more or less challenging to parse). 

In an effort for harm prevention, the dataset that is publicly available strips out information such as geography, violation type, or facility name that may actually prove useful for predictions (e.g., in the case of spatial spillovers). Perhaps there’s a way the authors can indicate a ‘trusted analyst’ position for others to be vetted to use the full dataset so as to incorporate such data a la Census Sworn Status? It’s also surprising to see that 74% of inspections yield violations, and that makes me wonder if there’s degrees of variation in compliance with inspections that one may want to take into account in subsequent analyses, if sufficient information is available, and/or if there are ways to subdivide/focus on a subset of the population. 

The linked dataset is also in Danish, which limits accessibility (there is an option in the upper right to convert to English but then it no longer remains on the same page. An English language link, if available, would enhance accessibility.  Else a note indicating the webpage is not in English would also be helpful. 

correctness: The largest challenges that seems to be plaguing their analysis is lack of variation in the dependent variable (e.g. 75% of the observations have some form of noncompliance). I do wonder if considering the distribution of compliance problems would yield more interesting and useful insights.
clarity: The paper is well written and easy to read. Figure 1 is also very useful for guiding the reader, as are the two possible use cases. 
relation to prior work: 
This work is contextualized and noted as different form other works in the content application (health and safety instead of food inspections, crime, or environmental enforcement). 

additional feedback: I'd invest in this paper by citing it and would be curious to see what further analysis results. Other countries have similar types of questions and problems, so having some model to draw upon and an understanding of which factors may be most strongly associated with violations could help guide their inquiries even when regulations vary.
review_title: Danish-language dataset on labor law compliance

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The authors release a Labour Inspection Checklists dataset that contains ~63k instances of inspections conduced by Norwegian Labour Inspection Authority (NLIA) during the period of January 2013 to June 2019. The dataset contains 575 features and 2 target variables. The features contain meta data on organizational or financial information about the company under inspection. The target variables are 1) Checklist inspection topic, 2) Is company compliant for all questions or non compliant for at-least one questions in the checklist. The dataset is based on actual daily operations of NLIA without any transformations.  To show how the authors envision this dataset as a useful dataset for ML community, the results from baseline experiments for prediction of target variables are discussed.  

The contributions from authors include curation of a dataset that can be used to make the inspections of HSE violations easier for governmental agencies that conduct labour inspections. ML can help such agencies with limited resources to better plan the inspections.
strengths: If the ML research on this dataset can show practically useful results, then it can be very useful for agencies that perform an important job of making sure that the employees work in safe working conditions. ML based recommendations can help inspection agencies in shortlisting high risk companies to inspect on priority and in quickly creating the right checklist of questions to evaluate during inspections.  
 
The authors promise to share dataset publicly after the paper review process. At this point, it is hard to comment on the accessibility of the dataset for wider research community. As per authors, the dataset is compliant to GDPR and Norwegian privacy protection laws. It is also made sure that the data does not contain privacy violating features by getting dataset reviewed by the management of inspection agency.
weaknesses: The paper as several weaknesses as listed below:

1. *Evaluation metrics:* The two category of experiments used to demonstrate the application of ML for predicting target variables are evaluated using metrics that dwell on hard labels, i.e. accuracy, precision and recall. The models used in experiments return soft scores and in order to convert soft scores to hard labels (0/1 or 0..n if multi class), a threshold is chosen as pivot point. The authors did not make it clear how threshold was chosen and in fact the pivot point is unknown. Seems like default threshold of 0.5 was chosen but 0.5 does not make sense for non calibrated models like GradientBoost. It is unknown, what is the desirable precision or desirable recall that can be used for tuning the threshold. Instead, for comparisons of different models, it would be better to first evaluate the models in terms of metrics that can use soft scores, like AUC (micro/macro) and accordingly show better performing models.
2. *Missing feature engineering details:* Missing values getting imputed by 0 limit ability for researcher to use their own imputation method. A researcher might want to impute with mean or median, or a -1 (depending on the range of values in the data). Was there any data transformation done on any features before passing to the models? How many features were categorical and how were they treated? Such details on feature engineering are missing.
3. *Feature Selection:* Much discussion is provided on feature selection but there is no result based on full set of features. In most of the chosen models, the use of regularization can inherently handle 575 features and remove ones that are less useful. As a result, feature selection step in many cases can turn out to be of minimal benefit. Also, if author believe feature selection to be an important step, then a discussion of top features impacting the target outcome is missing. 
4. *Lack of error bars:* The authors mention use of cross validation but error bars are not provided in results. If cross validation was not used, then details about train, validation and test splits is missing. It would have been better to add details on how the sizes of train, validation and test sets if used. 
5. *Lack of discussion on application of results:* The performance for both tasks using different models is low. Authors have not mentioned what is the target performance that could be of a practical help for agency. Is the current performance for both tasks practically useful? If yes, how does the agency plan to use the predictions? Moreover, CLSP is a multi class classification task and it is unclear which class is more predictable in the 10.6% accuracy achieved by DT model. 
6. *Dataset not scalable to other countries limiting its impact:* The dataset is curated by Norwegian Inspection agency and comprises of inspections performed in Norway. This limits the impact and usefulness of dataset to broader research community from other geographies.   
7. *Imbalanced vs. balanced*: 26%-74% class ratio is generally not required to be under/over sampled. Under/over sampling is generally done for highly imbalanced datasets (e.g. 1 in 1000 instances). Moreover, test sets are kept in original class ratio when under/over sampling is performed to provide real evaluation metrics that will mimic production scenario. Instead, if the class ratio is not 50-50, use of right evaluation metrics (like AUC instead of Accuracy) comes into play. 
8. *Data distribution analysis:* It would be useful to provide distribution analysis of different attributes present in the data. How many attributes were categorical, how many numeric, what is the relationship of different variables with target variable etc. 


correctness: The dataset is pulled from agency’s database that performs HSE inspections in Norway. The primary author work at the same agency and has done the management review to make sure dataset can be released publicly. The organizational information is taken from the time of inspection rather than most recent information, which makes sense for the task at hand. 

However, the dataset creation does not seem 100% correct. For example, the authors themselves imputed missing values with 0 which should be left to the user of the dataset to decide how to impute missing values. Similarly, the choice of metrics and evaluation methodology of baseline ML approaches does not seem to be correct, as also mentioned in the weakness section.  
clarity: The paper is clear to read and understand. 
relation to prior work: The related work in similar as well as different domains are clearly discussed.  
additional feedback: Please check following lines for corrections:

* Line 52: Mention January 2013 but in attached data set documentation, 1/1/2012 
* What (other) tasks could the dataset be used for?:  “The data set could be relevant for benchmarking ML-methods dealing with imbalanced” → Disagree that 74/26 is the level of imbalance that can be used in research of imbalanced datasets. In real world scenarios, we deal with imbalanced datasets with 1/1000 or even 1/10000 class ratios.
review_title: Labour Inspection dataset curated from inspections done for various industries in Norway during 2013 to 2019 

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The paper proposes a dataset of ~64K points for two tasks: selecting the best checklist for inspection and predicting non-compliance
strengths: The problem the paper attacks is an interesting one which has real, practical applications and could help workers.
weaknesses: While the dataset seems sound, some choices in the design of experimentation makes it so that the proposed baseline is weaker than it needs be, namely:
- For the checklist selection problem, it is not a 369 way classification problem, as not every checklist can be applied to every industry, which reduces the number of classes.
- The experiments used are simple statistical methods, which don't seem to have been selected with any particular reasoning behind them. Given the promises of DL in pretty much ever area, it feels like a severe limitation to not have at least a BERT or some pre-trained algorithm as a baseline.
- The feature selection method is done using a shortcut, where methods are evaluated only on DT and eliminated based on this result. Given the short running time of most of the algorithms, it would have been better to simply eliminate Sequential Selection and run every selection method on every algorithm. There’s also no report of the result without feature selection.
- The 5-cross validation seems like a good way to have results that people can't reproduce. The dataset is large enough to be split into a standard train/dev/test, which would allow practitioners to compare on the same test set. In the same vein, it is not clear how you fine-tune your algorithms given the absence of the test set. The methodology makes me believe that there is possibly peeking involved in the early stage (such as the feature selection)
- It would be beneficial to extend on the differences between the previously released dataset by the same authors (in 13, 14) and this one. One of the highlighted contribution is that this is a new dataset, but it seems like it's the same data with different preprocessing, which would make it new tasks on the data. While this doesn't change what is accomplished, it would bring more transparency to the paper.
correctness: The claims seem correct. Experiment design could be improved, as highlighted above.
clarity: A lot of acronyms are used, which can hinder the clarity of the paper. For example, it would be helpful to expand a bit on what SDG means since it feels central to the contribution of the paper. It would also be worthwhile to spend a little more time describing where the checklists come from. 
relation to prior work: It would be helpful to expand on the difference between the previously released dataset in 13,14 and this one. Is it the same data?
additional feedback: In the resubmission statement you state that you "translated" the dataset to English. This raises a bit of a concern as it is not mentionned in the paper.
review_title: LICD dataset, possibly new?

Meta-review: metareview: The paper proposes a dataset on labor inspections, where a set of curated features are used to predict inspection checklists and violations. The perspective is unique and fresh to the machine learning community, and the underlying goal of safe working environment is relevant to everyone. The reviewers mostly agree with this viewpoint, despite some pointed out the current results are not competent or some details/analysis are missing. This leaves room for future research, and the main contribution is the hard-to-obtain dataset and the new problem. The author also provided detailed rebuttals with a revised draft, addressing many review comments. In my opinion the dataset/problem contribution outweighs potential flaws in experiments, and I recommend the paper to be accepted.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept