Using word2vec to Analyze News Headlines and Predict Article Success
By: Charlene Chambliss
Tags: In Depth Analysis, Towards Data Science, Data Science, Data Visualization, Machine Learning

The distributions look fairly similar, but it’s a little hard to tell how similar when they’re all on different plots. Let’s try overlaying them all on one plot.

# Overlay each density curve on the same plot for closer comparison



fig, ax = plt.subplots(figsize=(12, 8))



for source, color in zip(source_names, source_colors):

sns.distplot(main_data.loc[main_data['Source'] == source]['SentimentTitle'],

ax=ax, hist=False, label=source, color=color)

ax.set_xlabel('')



plt.xlim(-0.75, 0.75)

plt.show()

We see that the sources’ Sentiment distributions for article titles are very similar — it doesn’t look like any one source is an outlier in terms of positive or negative titles. Instead, all 12 of the most common sources have distributions centered around 0 with modestly sized tails. But does that tell the full story? Let’s take one more look at the numbers:

# Group by Source, then get descriptive statistics for title sentiment

source_info = main_data.groupby('Source')['SentimentTitle'].describe() # Recall that `source_names` contains the top 12 sources

# We'll also sort by highest standard deviation

source_info.loc[source_names].sort_values('std', ascending=False)[['std', 'min', 'max']]

WSJ has both the highest standard deviation and the largest range.

We can see at a glance that WSJ has both the highest standard deviation and the largest range, with the lowest minimum sentiment compared to any other top source. This suggests that WSJ could be unusually negative in terms of its article titles. To verify this rigorously would require a hypothesis test, which is beyond the scope of this post, but it’s an interesting potential finding and future direction.

Popularity Prediction

Our first task in preparing the data for modeling is to rejoin the document vectors with their respective titles. Thankfully, when we were preprocessing the corpus, we processed the corpus and titles_list simultaneously, so the vectors and the titles they represent will still match up. Meanwhile, in main_df , we have dropped all of the articles that had -1 popularity, so we'll need to drop the vectors that represent those article titles.

Training a model on these enormous vectors as-is will not be possible on this computer, but we’ll see what we can do with a little dimension reduction. I’ll also engineer a new feature from publish date: “DaysSinceEpoch”, which is based on Unix time (read more here).

import datetime



# Convert publish date column to make it compatible with other datetime objects



main_data['PublishDate'] = pd.to_datetime(main_data['PublishDate'])



# Time since Linux Epoch

t = datetime.datetime(1970, 1, 1)



# Subtract this time from each article's publish date

main_data['TimeSinceEpoch'] = main_data['PublishDate'] - t



# Create another column for just the days from the timedelta objects

main_data['DaysSinceEpoch'] = main_data['TimeSinceEpoch'].astype('timedelta64[D]')



main_data['TimeSinceEpoch'].describe()

As we can see, all of these articles were published within about 250 days of each other.

from sklearn.decomposition import PCA



pca = PCA(n_components=15, random_state=10)



# as a reminder, x is the array with our 300-dimensional vectors

reduced_vecs = pca.fit_transform(x) df_w_vectors = pd.DataFrame(reduced_vecs)



df_w_vectors['Title'] = titles_list # Use pd.concat to match original titles with their vectors

main_w_vectors = pd.concat((df_w_vectors, main_data), axis=1)



# Get rid of vectors that couldn't be matched with the main_df

main_w_vectors.dropna(axis=0, inplace=True)

Now we need to drop non-numeric and non-dummy columns so we can feed the data to a model. We’ll also apply scaling to the DaysSinceEpoch feature, since it is wildly larger in magnitude compared to the reduced word vectors, sentiment, etc.

# Drop all non-numeric, non-dummy columns, for feeding into the models

cols_to_drop = ['IDLink', 'Title', 'TimeSinceEpoch', 'Headline', 'PublishDate', 'Source']



data_only_df = pd.get_dummies(main_w_vectors, columns = ['Topic']).drop(columns=cols_to_drop)



# Standardize DaysSinceEpoch since the raw numbers are larger in magnitude

from sklearn.preprocessing import StandardScaler



scaler = StandardScaler()



# Reshape so we can feed the column to the scaler

standardized_days = np.array(data_only_df['DaysSinceEpoch']).reshape(-1, 1)

data_only_df['StandardizedDays'] = scaler.fit_transform(standardized_days)



# Drop the raw column; we don't need it anymore

data_only_df.drop(columns=['DaysSinceEpoch'], inplace=True)



# Look at the new range

data_only_df['StandardizedDays'].describe()

# Get Facebook data only

fb_data_only_df = data_only_df.drop(columns=['GooglePlus', 'LinkedIn']) # Separate the features and the response

X = fb_data_only_df.drop('Facebook', axis=1)

y = fb_data_only_df['Facebook']



# 80% of data goes to training

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 10)

Let’s run a non-optimized XGBoost on the data and see how it works out-of-the-box.

from sklearn.metrics import mean_squared_error



# Instantiate an XGBRegressor

xgr = xgb.XGBRegressor(random_state=2)



# Fit the classifier to the training set

xgr.fit(X_train, y_train)



y_pred = xgr.predict(X_test)



mean_squared_error(y_test, y_pred)

Underwhelming results, to say the least. Can we improve this performance with hyperparameter tuning? I’ve pulled in and repurposed a hyperparameter tuning grid from this Kaggle article.

from sklearn.model_selection import GridSearchCV



# Various hyper-parameters to tune

xgb1 = xgb.XGBRegressor()

parameters = {'nthread':[4],

'objective':['reg:linear'],

'learning_rate': [.03, 0.05, .07],

'max_depth': [5, 6, 7],

'min_child_weight': [4],

'silent': [1],

'subsample': [0.7],

'colsample_bytree': [0.7],

'n_estimators': [250]}



xgb_grid = GridSearchCV(xgb1,

parameters,

cv = 2,

n_jobs = 5,

verbose=True)



xgb_grid.fit(X_train, y_train)

According to xgb_grid , our best parameters were as follows:

{'colsample_bytree': 0.7, 'learning_rate': 0.03, 'max_depth': 5, 'min_child_weight': 4, 'n_estimators': 250, 'nthread': 4, 'objective': 'reg:linear', 'silent': 1, 'subsample': 0.7}

Try again with the new parameters:

params = {'colsample_bytree': 0.7, 'learning_rate': 0.03, 'max_depth': 5, 'min_child_weight': 4,

'n_estimators': 250, 'nthread': 4, 'objective': 'reg:linear', 'silent': 1, 'subsample': 0.7}



# Try again with new params

xgr = xgb.XGBRegressor(random_state=2, **params)



# Fit the classifier to the training set

xgr.fit(X_train, y_train)



y_pred = xgr.predict(X_test)



mean_squared_error(y_test, y_pred)

It’s better by around 35,000, but I’m not sure that’s saying a whole lot. We might infer, at this point, that the data in its current state seems insufficient for this model to perform. Let’s see if we can improve it with a little more feature engineering: we’ll train some classifiers to separate the two main groups of articles: Duds (0 or 1 share) vs. Not Duds.

The idea is that if we can give the regressor a new feature (the probability that the article will have extremely low shares), it may perform more favorably on predicting highly-shared articles, thus lowering the residual values for those articles and reducing mean squared error.

Detour: Detect Dud Articles

From the log-transformed plots we made earlier, we can note that in general, there are 2 chunks of articles: 1 cluster at 0, and another cluster (the long tail) going from 1 onwards. We can train a few classifiers to identify whether the article will be a “dud” (be in the 0–1 shares bin), and then use the predictions of those models a feature for the final regressor, which will predict probability. This is called model stacking.

# Define a quick function that will return 1 (true) if the article has 0-1 share(s)

def dud_finder(popularity):

if popularity <= 1:

return 1

else:

return 0 # Create target column using the function

fb_data_only_df['is_dud'] = fb_data_only_df['Facebook'].apply(dud_finder)

fb_data_only_df[['Facebook', 'is_dud']].head()

# 28% of articles can be classified as "duds"

fb_data_only_df['is_dud'].sum() / len(fb_data_only_df)

Now that we have our dud feature made, we’ll initialize the classifiers. We’ll use a Random Forest, an optimized XGBClassifier, and a K-Nearest Neighbors classifier. I will leave out the part where I tune the XGB, since it looks essentially the same as the tuning we did earlier.

from sklearn.ensemble import RandomForestClassifier

from sklearn.neighbors import KNeighborsClassifier

from sklearn.model_selection import train_test_split X = fb_data_only_df.drop(['is_dud', 'Facebook'], axis=1)

y = fb_data_only_df['is_dud'] # 80% of data goes to training

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 10) # Best params, produced by HP tuning

params = {'colsample_bytree': 0.7, 'learning_rate': 0.03, 'max_depth': 5, 'min_child_weight': 4,

'n_estimators': 200, 'nthread': 4, 'silent': 1, 'subsample': 0.7} # Try xgc again with new params

xgc = xgb.XGBClassifier(random_state=10, **params)

rfc = RandomForestClassifier(n_estimators=100, random_state=10)

knn = KNeighborsClassifier() preds = {}

for model_name, model in zip(['XGClassifier', 'RandomForestClassifier', 'KNearestNeighbors'], [xgc, rfc, knn]):

model.fit(X_train, y_train)

preds[model_name] = model.predict(X_test)

Test the models, get the classification reports:

from sklearn.metrics import classification_report, roc_curve, roc_auc_score for k in preds:

print("{} performance:".format(k))

print()

print(classification_report(y_test, preds[k]), sep='

')

The top performance in terms of f1-score came from the XGC, followed by the RF and finally the KNN. However, we can also note that the KNN actually did the best job in terms of recall (successfully identifying duds). This is why model stacking is valuable — sometimes even an otherwise excellent model like XGBoost can underperform on tasks like this one, where evidently the function to be identified can be locally approximated. Including the KNN’s predictions should add some much-needed diversity.

# Plot ROC curves

for model in [xgc, rfc, knn]:

fpr, tpr, thresholds = roc_curve(y_test, model.predict_proba(X_test)[:,1])

plt.plot([0, 1], [0, 1], 'k--')

plt.plot(fpr, tpr)



plt.xlabel('False Positive Rate')

plt.ylabel('True Positive Rate')

plt.title('ROC Curves')

plt.show()

Popularity Prediction: Round 2

Now we can average out the probability predictions from the three classifiers, and use it as a feature for the regressor.

averaged_probs = (xgc.predict_proba(X)[:, 1] +

knn.predict_proba(X)[:, 1] +

rfc.predict_proba(X)[:, 1]) / 3 X['prob_dud'] = averaged_probs

y = fb_data_only_df['Facebook']

This is followed by another round of HP tuning with the new feature included, which I’ll leave out. Let’s see how we do on performance:

xgr = xgb.XGBRegressor(random_state=2, **params) # Fit the classifier to the training set

xgr.fit(X_train, y_train) y_pred = xgr.predict(X_test) mean_squared_error(y_test, y_pred)

Uh oh! This performance is essentially the same as it was before we even did any model stacking. That said, we can keep in mind that the MSE as an error measurement tends to overweight outliers. In fact, we can also calculate the mean absolute error (MAE), which is used for assessing performance on data with significant outliers. In mathematical terms, the MAE calculates the l1 norm, essentially the absolute value, of the residuals, rather than the l2 norm used by the MSE. We can compare the MAE with the square root of the MSE, also known as the root mean squared error (RMSE).

mean_absolute_error(y_test, y_pred), np.sqrt(mean_squared_error(y_test, y_pred))

The mean absolute error is only about 1/3 of the RMSE! Perhaps our model is not as bad as we might have initially thought.

As a final step, let’s take a look at each feature’s importance according to the XGRegressor:

for feature, importance in zip(list(X.columns), xgr.feature_importances_):

print('Model weight for feature {}: {}'.format(feature, importance))

Aha! prob_dud was found to be the most important feature.

Neat! Our model found prob_dud to be the most important feature, and our custom StandardizedDays feature was the second most important. (Features 0 through 14 correspond to the reduced title embedding vectors.)

Even though overall performance didn’t improve through this round of model stacking, we can see that we did successfully capture an important source of variability in the data, which the model picked up on.

If I were to continue expanding on this project to make the model more accurate, I’d probably consider augmenting the data with outside data, including Source as a variable via binning or hashing, running the models on the original 300-dimensional vectors, and using the “time-sliced” data (a companion dataset to this data) of each article’s popularity at various time points to predict the final popularity.

If you found this analysis interesting, please feel free to use the code, and to expand on it further! The notebook is here (note that some of the cells may be in a slightly different order than they were presented here), and the original data used for this project is here.