Predicting Fake job postings — Part 2 (Predictive Analysis) | by Sharad Jain | Towards Data Science

In Part 1 of this two-part series, I did a detailed exploratory analysis of how fake job postings are different from genuine ones. Below is a link to Part 1 and I would highly recommend you read that first before moving forward!
towardsdatascience.com
Fake job postings were mostly targeted towards Full-time roles, which had a minimum requirement of a Bachelor’s degree and a Mid-Senior level work experience.These postings were mainly directed towards individuals seeking jobs in the technology sector.45% of these postings gave jobs to candidates without even a single question asked (A vital indicator of a fake posting).The postings in which had no screening process were mostly for Entry level jobs.The fake postings were mostly in the United States, followed by the United Kingdom, Canada, and India.After creating a word cloud, it was found that the job postings had similar behavioral content, but the genuine ones were more specific to the role.
Fake job postings were mostly targeted towards Full-time roles, which had a minimum requirement of a Bachelor’s degree and a Mid-Senior level work experience.
These postings were mainly directed towards individuals seeking jobs in the technology sector.
45% of these postings gave jobs to candidates without even a single question asked (A vital indicator of a fake posting).
The postings in which had no screening process were mostly for Entry level jobs.
The fake postings were mostly in the United States, followed by the United Kingdom, Canada, and India.
After creating a word cloud, it was found that the job postings had similar behavioral content, but the genuine ones were more specific to the role.
Based on these insights, we now know that it is possible to find out which job postings are fake and which are not. But in these unprecedented times, where hundreds of individuals are being laid off every day, job seekers are desperate. The scammers are using this desperation to put out more and more fake job advertisements. Hence, we need more of these algorithms and tools being used on job search websites like LinkedIn, Glassdoor and Indeed so that these fake postings are filtered out, and the job-seekers only see the genuine ones.
In the following sections, I will be using machine learning techniques on the data used last time and predict the fake job postings from the genuine ones.
So, the data we have used so far in this analysis was compiled by the University of the Aegean, Laboratory of Information & Communications Systems Security (http://emscad.samos.aegean.gr/). This dataset contains 800 fak job descriptions.
I have defined the variables which are a part of the dataset initially in Part 1 of the series. I will be mentioning which variables out of the complete list. I will be using for this analysis in the next section.
I imported the data on the Jupyter Notebook on my system and worked on Python 3. Since I want this part to be more on insights than code, I have not attached any code gists here but anyone interested in looking at the data cleaning code, feel free to check out my GitHub repository — https://github.com/sharad18/Fake_Job_Posting
The initial dataset contained 18,000 postings. After cleaning, the new data has ~11,000 postings.
The subset of variables which I will be using for the predictive analysis will be —
Title: Title of the job posting.Description: Job description + Company profile + RequirementsTelecommuting: Tru for telecommuting positions.Has_company_logo: True if the company logo is present.Has_questions: True if screening questions are present.Employment_type: Full-time, Part-time, Contract, etc.Required_experience: Executive, Entry-level, Intern, etc.Required_education: Doctorate, Master’s degree, Bachelor’s, etc.Industry: Automotive, IT, Health care, Real estate, etc.Function: Consulting, Engineering, Research, Sales, etc.Fraudulent (Target Variable): 1 if fake posting else 0.City: The city mentioned in the job posting.Country_name: Name of the country mentioned in the job posting.
Title: Title of the job posting.
Description: Job description + Company profile + Requirements
Telecommuting: Tru for telecommuting positions.
Has_company_logo: True if the company logo is present.
Has_questions: True if screening questions are present.
Employment_type: Full-time, Part-time, Contract, etc.
Required_experience: Executive, Entry-level, Intern, etc.
Required_education: Doctorate, Master’s degree, Bachelor’s, etc.
Industry: Automotive, IT, Health care, Real estate, etc.
Function: Consulting, Engineering, Research, Sales, etc.
Fraudulent (Target Variable): 1 if fake posting else 0.
City: The city mentioned in the job posting.
Country_name: Name of the country mentioned in the job posting.
Let’s start working on this cleaned data!
I imported the following libraries for now which are the essential libraries anyone needs to perform analysis in Python,
import pandas as pdimport seaborn as snsimport matplotlib.pyplot as plt%matplotlib inlineimport numpy as nppd.set_option('display.max_columns', None)
Let’s look at the data,
df = pd.read_csv('Clean_data.csv')df.head()
Now, we try to study the column ‘description’ in detail. From the cleaned data file I imported, the ‘description’ column is the combination of ‘description’ + ‘requirements’ and ‘company profile.’ An example of how the data in ‘description’ looks like —
Next, we will try to look at the Top 20 most common words being used in both fake postings as well as good job postings.
import spacynlp = spacy.load('en_core_web_lg')import base64import stringimport refrom collections import Counterfrom nltk.corpus import stopwordsstopwords = stopwords.words('english')punctuations = string.punctuationdef cleanup_text(docs, logging = False):    texts = []    for doc in docs:        doc = nlp(doc, disable = ['parser', 'ner'])        tokens = [tok.lemma_.lower().strip() for tok in doc if tok.lemma_ != '-PRON-']        tokens = [tok for tok in tokens if tok not in stopwords and tok not in punctuations]        tokens = ' '.join(tokens)        texts.append(tokens)return pd.Series(texts)
For fraudulent job postings,
Fraud_1 = [text for text in df1[df1['fraudulent'] == 1]['description']]Fraud_1_clean = cleanup_text(Fraud_1)Fraud_1_clean = ' '.join(Fraud_1_clean).split()Fraud_1_counts = Counter(Fraud_1_clean)Fraud_1_common_words = [word[0] for word in Fraud_1_counts.most_common(20)]Fraud_1_common_counts = [word[1] for word in Fraud_1_counts.most_common(20)]fig = plt.figure(figsize = (20, 10))pal = sns.color_palette("cubehelix", 20)sns.barplot(x = Fraud_1_common_words, y = Fraud_1_common_counts, palette=pal)plt.title('Most Common Words used in Fake job postings')plt.ylabel("Frequency of words")plt.xlabel("Words")plt.show()
For genuine job postings,
Fraud_0 = [text for text in df1[df1['fraudulent'] == 0]['description']]Fraud_0_clean = cleanup_text(Fraud_0)Fraud_0_clean = ' '.join(Fraud_0_clean).split()Fraud_0_counts = Counter(Fraud_0_clean)Fraud_0_common_words = [word[0] for word in Fraud_0_counts.most_common(20)]Fraud_0_common_counts = [word[1] for word in Fraud_0_counts.most_common(20)]fig = plt.figure(figsize = (20, 10))pal = sns.color_palette("cubehelix", 20)sns.barplot(x = Fraud_0_common_words, y = Fraud_0_common_counts, palette=pal)plt.title('Most Common Words used in Genuine job postings')plt.ylabel("Frequency of words")plt.xlabel("Words")plt.show()
In both the figures above, most frequent words are almost alike, and it is tough to differentiate between the two. Hence, the variable ‘description’ alone cannot help us with predicting, and therefore the complete set of variables is significant. One more noteworthy thing to point out is, when an individual sees a job posting on the internet, these additional features which I have mentioned in the ‘Data Cleaning’ section are not directly provided. These variables have only been acquired after a job posting was reported as fraudulent. In such a case, it becomes imperative that if one finds such a fake posting, they inform the concerned authorities as well as to other job-seekers in their network.
Before splitting the data, we need to make some final changes to the data, after which we will be splitting it. The ‘description’ column had to be cleaned into tokens. This is done with the help of spacy and nltk libraries. In the following piece of code, the ‘description’ column has been converted into tokens, and those tokens have been used to create one hot column using sklearn’s CountVectorizer. Also, categorical columns like ‘employment_type,’ ‘required_eduacation,’ ‘required_experience,’ ‘industry,’ and ‘function’ have been converted into one-hot vectors.
STOPLIST = set(stopwords.words('english') + list(ENGLISH_STOP_WORDS))SYMBOLS = " ".join(string.punctuation).split(" ")def tokenizetext(sample):    text = sample.strip().replace("\n", " ").replace("\r", " ")    text = text.lower()    tokens = parser(text)    lemmas = []    for tok in tokens:        lemmas.append(tok.lemma_.lower().strip() if tok.lemma_ != "-PRON-" else tok.lower_)    tokens = lemmas    tokens = [tok for tok in tokens if tok not in STOPLIST]    tokens = [tok for tok in tokens if tok not in SYMBOLS]    return tokensvectorizer = CountVectorizer(tokenizer = tokenizetext, ngram_range = (1,3), min_df = 0.06)vectorizer_features = vectorizer.fit_transform(df1['description'])vectorized_df = pd.DataFrame(vectorizer_features.todense(), columns = vectorizer.get_feature_names())df_final = pd.concat([df1, vectorized_df], axis = 1)df_final.drop('description', axis = 1, inplace = True)df_final.dropna(inplace=True)columns_to_1_hot = ['employment_type', 'required_experience', 'required_education', 'industry', 'function']for column in columns_to_1_hot:    encoded = pd.get_dummies(df_final[column])    df_final = pd.concat([df_final, encoded], axis = 1)columns_to_1_hot += ['title', 'city', 'country_name']df_final.drop(columns_to_1_hot, axis = 1, inplace = True)
Next, we split the data into train and test —
target = df_vectorized['fraudulent']features = df_vectorized.drop('fraudulent', axis = 1)X_train, X_test, y_train, y_test = train_test_split(features, target, test_size = 0.1, stratify = target, random_state=42)print (X_train.shape)print (y_train.shape)print (X_test.shape)print (y_test.shape)
We get the following output —
(10144, 857)(10144,)(1128, 857)(1128,)
We have everything place now. Let’s start with machine learning!
Libraries we will use in subsequent sections —
from sklearn.model_selection import GridSearchCVfrom sklearn.linear_model import LogisticRegressionfrom sklearn.neighbors import KNearestNeighborsfrom sklearn.svm import SVCfrom sklearn.ensemble import RandomForestClassifierfrom sklearn.neural_network import MLPClassifierfrom sklearn.metrics import accuracy_score, confusion_matrix, classification_report, roc_auc_score
log_reg = LogisticRegression()c_values = [0.00001, 0.0001, 0.001, 0.01, 0.1, 1, 10, 100, 1000, 10000]penalty_options = ['l1', 'l2']param_grid = dict(C = c_values, penalty = penalty_options)grid_tfidf = GridSearchCV(log_reg, param_grid = param_grid, cv = 10, scoring = 'roc_auc', n_jobs = -1, verbose=1)grid.fit(X_train, y_train)log_reg_pred = grid.predict(X_test)print (roc_auc_score(y_test, log_reg_pred))print (classification_report(y_test, log_reg_pred))
We get the following results — Out of Sample roc_auc score = 0.7795 and for classification report we get —
Predictions with the most basic model, Logistic Regression, gives us a 0.7795 ROC-AUC score, which is a good score given the imbalance of the data. In subsequent sections, we’ll see more sophisticated models.
knn = KNeighborsClassifier()k_range = list(np.arange(2, 23, 2))param_grid_knn = dict(n_neighbors = k_range)print (param_grid_knn)grid_knn = GridSearchCV(knn, param_grid_knn, cv = 10, scoring = 'roc_auc', n_jobs = -1, verbose = 1)grid_knn.fit(X_train, y_train)knn_pred = grid_knn.predict(X_test)print (roc_auc_score(y_test, knn_pred))print (classification_report(y_test, knn_pred))
Out of Sample ROC-AUC score for KNN — 0.5995. For classification report we get —
From the image on the left, we can see that KNN performed very poorly as compared to Logistic Regression.
svc = SVC()kernel = ['linear', 'rbf']param_grid_knn = dict(kernel = kernel)print (param_grid_knn)grid_svc = GridSearchCV(svc, param_grid_knn, cv = 10, scoring = 'roc_auc', n_jobs = -1, verbose = 2)grid_svc.fit(X_train, y_train)svc_pred = grid_svc.predict(X_test)print (roc_auc_score(y_test, svc_pred))print (classification_report(y_test, svc_pred))
The out of sample score reported here is 0.8195 ~ 0.82. The classification report is as follows —
SVC has the best score yet with 0.82, which is significantly better than the last two methods.
rf = RandomForestClassifier()n_estimators_range = [1, 2, 4, 8, 16, 32, 64, 100, 200]param_grid_rf = dict(n_estimators = n_estimators_range)grid_rf = GridSearchCV(rf, param_grid_rf, cv = 10, scoring = 'roc_auc', n_jobs = -1, verbose = 1)grid_rf.fit(X_train, y_train)print (grid_rf.best_score_)print (grid_rf.best_params_)rf_pred = grid_rf.predict(X_test)print (roc_auc_score(y_test, rf_pred))print (classification_report(y_test, rf_pred))
For the Random Forest model, the ROC-AUC score reported is 0.74. The classification report is as follows —
To be honest, I expected Random Forest to perform better than SVC, but so far, SVC has the best result.
mlp = MLPClassifier(solver = 'sgd', activation = 'relu', hidden_layer_sizes = (100, 50, 30), max_iter = 1000)mlp.fit(X_train, y_train)mlp_pred = mlp.predict(X_test)print (roc_auc_score(y_test, mlp_pred))print (classification_report(y_test, mlp_pred))
ROC-AUC score, in this case, is 0.7786. And the classification report —
The MLP Classifier with ‘sgd’ solver performs slightly better than the Random Forest model but is still worse than what SVC reported.
mlp = MLPClassifier(solver = 'adam', activation = 'relu', hidden_layer_sizes = (100, 50, 30), max_iter = 1000)mlp.fit(X_train, y_train)mlp_pred = mlp.predict(X_test)print (roc_auc_score(y_test, mlp_pred))print (classification_report(y_test, mlp_pred))
The out of sample ROC-AUC score reported for this model is 0.8595 ~ 0.86. The classification report is as follows —
This is the highest score yet reported, and based on this model, we will be able to predict correctly if a job posting is fake 86% of the time.
From the above image, we can see that Sklearn’s MLP Classifier based on Neural Network with ‘adam’ optimizer performs the best followed by the same model with ‘sgd’ solver and Support Vector Machine Classifier. So based on MLP Classifier’s out of sample performance metric, ROC-AUC score, we can safely say that we can predict if a job posting is fake or not 86% of the time.
This predictive analysis is very much useful for job search websites like LinkedIn and Glassdoor, which can help them filter such fak job postings.
Currently, we live in times which none of us ever expected. Coronavirus has not only brought health emergencies to nations but also accelerated an impending recession. Many employees are being laid off every day, and the demand for jobs is way higher than the number of posts available in the market.
Through this series of articles, I have tried to put forward an issue that is creeping through the job market. Turmoil and chaos are the perfect proponents for scammers, and currently, cyber scam attacks are on the rise. In the previous part, I provided a detailed analysis of how to differentiate between a fake posting and a genuine one and what are the characteristics of a fraudulent job posting. In this part, I have provided a detailed analysis of how we can apply machine learning to predict the occurrences of such fake postings.
As I have mentioned throughout the two parts, rather than any model or analysis, it is the responsibility of us as users of the internet and job-seekers to let the authorities and our network know if we come across such a fake job posting.
Thank you for reading the two parts. I wish everyone a great day ahead & please maintain social distance so that we can get over such times faster and more efficiently!
Working on this project was a great learning experience for me. Hopefully, you all found it helpful and informative.
The entire code for this article is there on my GitHub repository for this analysis — https://github.com/sharad18/Fake_Job_Posting. I would love to hear your feedback in the comments section.
You can also connect with me on LinkedIn — https://www.linkedin.com/in/sharad-jain/