Case Study on SafeCity: Understanding Diverse Forms of Sexual Harassment Personal Stories
By: Subratho Das
Tags: Data Science, Deep Learning, Sexual Harassment, Machine Learning, Case Study

Case Study on SafeCity: Understanding Diverse Forms of Sexual Harassment Personal Stories Subratho Das Sep 7·7 min read

Content

Introduction Business Problem Mapping to ML problem Understanding the Data EDA Feature Engineering Model Explanation Results, Deployments Conclusions and Future Work References

1. Introduction

The idea of this case study came when I started reading this research paper. According to my understanding, the high level idea discussed in the research paper is to understand about the sexual harassment and to analyze it through nlp solutions because manually sorting information that can be useful to understand about harassment is an arduous task. This was motivated from campaign like #MeeToo (a social media campaign to spread awareness about sexual harassment issue) and focuses on automatically classifying vast amount of harassment data found on social media. Understanding about these harassments will also help in spreading awareness, increase in understanding when these harassment happens or who is responsible for them or on what location these harassment are taking place which will allow authorities for faster action.

But our case study focuses more on Machine learning models and we will try to beat the metrics score that was achieved by DL models in paper.

2. Business Problem

The business problem for our experiment is the task of automatically categorizing and analyzing various forms of sexual harassment, based on stories shared on the online forum SafeCity.

3. Mapping to ML problem

Since our problem is single label binary classification task. We will be using an accuracy metric same as paper because using this metric we can compare our scores with the paper.

4. Understanding the Data

The dataset is gathered from SafeCity, the largest publicly available online forum for reporting sexual harassment. But one can also find the dataset here. The dataset can be used for single label classification or multilabel classification and it has only one feature that is a text description about the sexual harassment and category associated with it which are ogling, groping, commenting. For our purpose we will be using the approach of single label classification. In this classification approach we have three datasets, one dataset will belong to ogling, one groping and one commenting, target variable associated with each of these dataset will consist 0 or 1. Single label models were evaluated using accuracy metric.

5. EDA

The most important EDA would be analyzing the word frequency to set an upper bound and lower bound when using some vectorization techniques to convert text feature to numerical feature. Another is knowing whether the words distribution accross train, cv and test dataset are same or not.

While doing EDA for word frequency, 100th percentile came somewhere around 6000 and after analyzing the words that have frequency over 1100, all of the words were articles or grammars. So this actually means we have to remove these words in preprocessing stages.

Currently our dataset is already divided into train, cv and test and after analyzing the word cloud in train, cv and test it was found that the words are almost similar across all the csv files. We will be also performing stemming since words across different datasets for the same class label have suffix added into it.

Upper and lower bound in word frequency

Image by author

Lets analyse the word frequency in our corpus to choose a lower and upper bound when using tfidf or bow. From above analysis I can easily see that my 90 percentile is 10 and then there is a huge difference between 90th percentile and 100th percentile, this also means that 90 percentage of my word frequency are less than 10.

One more point is that my 60 percentile of words occurs once, keeping lower bound for minimum frequency 1 makes sense, but we can also try with 2 or 4 as min freq.

We have do more analysis to find an upper bound.

Image by author

Now the word frequency is increasing slowly but still I can see there is a steeper incline between 99th percentile and 100th percentile. Lets do more analysis between 99th percentile and 100th percentile to achieve a upper bound threshold value.

Image by author

Image by author

Here the inclination has become smoother from 99.6 to 100 percentile values, therefore we can take 1404 as our upper bound. This also indicates toward the fact that for 99.9 percentage of words, there frequency is less than and equal to 1404.

Image by author

Further analysis of between 99.90 and 99.99 shows that at each step the percentile values changes with good difference value. We can also experiment with these values as upper bound.

Commenting wordcloud

Train Dataset: Image by author.

CV dataset: Image by author

Test Dataset: Image by author

We can see in above Images that word distribution across the different sets of dataset for commenting are almost same.

Ogling wordcloud

Train: Image by author

CV: Image by author

Test: Image by author

Distribution of words across all train, cv and test are almost similar, so this actually means we have similar kind of examples in cv as well as test.

Groping wordcloud

Train: Image by author

CV: Image by author

Test: Image by author

Distribution of words in groping also across all train, cv and test are almost similar so this actually means we have similar kind of examples in cv as well as test.

6. Feature Engineering

Apply these techniques directly on raw text. You can also refer here to know about these preprocessing steps.

Char count (FE): total number of characters in complete text. Word count (FE): total number of words in the complete text. Word density (FE): average length of the words used in the text. Punctuation count (FE): total number of punctuation marks in the essay. Upper case word count (FE): total number of upper count words in the essay

Now we can start cleaning our text, the stemming techniques which is discussed below will be kept optional to check which combination of these techniques yield us a best result.

Stopword: Remove all the stopwords from text and also convert all words to lowercase, make sure you do not remove “not” word. Stemming: Apply Snowball stemming over the text.

7. Model Explanation

For this we will be not applying any feature engineering (FE) techniques, we will be only applying techniques like stopword removal and stemming.

We are going to use Logistic regression and light gbm modelling techniques for our experiment.

Commenting

Image by author

Groping

Image by author

Ogling

Image by author

8. Results

Most of the model algorithms are performing well, if compared to scores with research paper scores, we are able to beat most of the models except CNN-RNN.

Image taken from paper

By doing modelling analysis we can easily see that using stemming is not yielding us good results.

So now we start modelling using lightGBM as our model. Here we applied all the feature engineering techniques and we did removed the stopwords also.

Commenting

Image by author

Groping

Image by author

Ogling

From the above scores we can see that we are able to beat the scores achieved by CNN-RNN which is discussed in paper for Groping and commenting. As for ogling we are still behind but we are able to beat other 6 modeling algorithms scores for ogling as discussed in the research paper.

9. Conclusion

In the above experiment we saw how different ML algorithms can be used for modeling with simple feature engineering techniques. You could use SHAP or LIME analysis to increase the explainability of models.

10. References