Multi-Class Text Classification with SKlearn and NLTK in python| A Software Engineering Use Case | by Nasir Safdari | Towards Data Science

Recently, I worked on a software engineering research project. one of the main objectives of the project was to understand the focus areas of work in the development teams. when the size of a software project becomes large, managing the workflow and the development process is more challenging. therefore, it is essential for the management team and lead developers to understand the type of work that is carried out by the software developers.
In more simple words, any developer at any time during different phases of the project can write pieces of code that aims to implement one of the followings:
Adding new features
Design improvements
Bug fixing
improving non-functional requirements
Now one might ask, are the above four categories the only types of software development activities done by the developers? and the answer is: No. the goal here is to come up with some general categories of work that encompasses the majority of development tasks.
Understanding how the work of development team is distributed regarding the mentioned four categories, can help the management to make better decisions on managing the growth of the software while continuously developing its functionalities. for example, if most of the work of development team is done toward bug-fixing, the management can take necessary actions to prevent faults and defects in the software and provide guidelines to the lead developers to pay more attention to the quality.
and the answer is: we can look at the commit messages; developers provide a commit message along with every commit they make to the repository. These commit messages are usually written using natural language, and generally convey some information about the commit they represent. we can build a classifier using some labeled data and then automatically classify future commits.
we collected around 4000 commit messages from several open source Java projects on github. we picked commit messages using an advanced search criteria based on specific key words for each category. then we assigned each category of commit messages to experienced developers to check commits one by one if they belong to their categories. after that, we cross checked the process by exchanging categories of commit messages among the developers to mitigate the subjectivity involved with manual labeling. finally, we ended up with almost 3400 commit messages including 282 commits that belonged to no category; we decided to add that as a label in the classification task so later on we can detect similar commit messages.
Let’s have a look at the data:
import pandas as pdimport reimport numpy as npimport matplotlib.pyplot as pltfrom nltk.corpus import stopwordsfrom nltk.stem import PorterStemmerfrom sklearn.feature_extraction.text import TfidfVectorizerfrom sklearn.pipeline import Pipelinefrom sklearn.metrics import classification_report, confusion_matrixfrom sklearn.model_selection import train_test_splitfrom sklearn.feature_selection import SelectKBest, chi2from sqlite3 import Errorfrom sklearn.ensemble import RandomForestClassifierimport sqlite3import pickle%matplotlib inlinetry:    conn = sqlite3.connect("training_V2.db")except Error as e:    print(e)#reading the data from the table that contains the labels df = pd.read_sql_query('SELECT * FROM filtered', conn)df.drop(['id'], 1, inplace=True)df.head()
as you can see, we imported so many packages and we are going to use all of them to achieve our goal. looking at the top five rows of the dataframe, we can see that it has only two columns: text (the commit messages) and class (the labels).
1 represents commit messages for bug fixing.
2 represents commits that belong to no category.
3 represents commit messages for design improvement.
4 represents commit messages for adding new features.
5 represents commit messages for improving non-functional requirements.
lets have a look at their distribution:
df.groupby('class').text.count().plot.bar(ylim=0)plt.show()
it is natural to have a little bit of imbalance in the dataset since the commits were collected randomly from random open source projects.
now lets build our classifier. but hold on... our data is in natural text but it needs to be formatted into a columnar structure in order to work as input to the classification algorithms.
here is how it works:
first, we remove the punctuation, numbers, and stop words from each commit message.
second, all the words are converted to lower case and then stemmed using the Porter Stemmer in the NLTK package. The goal of stemming is to reduce the number of inflectional forms of words appearing in the commit; it will cause words such as “performance” and “performing” to syntactically match one another by reducing them to their base word “perform”. This helps decrease the size of the vocabulary space and improve the volume of the feature space in the corpus.
Finally, each corpus is transformed into vector space model (VSM) using the tf-idf vectorizer in Python’s SKlearn package to extract the features.
all of this is done in just few lines of code and that is the beauty of python.
stemmer = PorterStemmer()words = stopwords.words("english")df['cleaned'] = df['text'].apply(lambda x: " ".join([stemmer.stem(i) for i in re.sub("[^a-zA-Z]", " ", x).split() if i not in words]).lower())
vectorizer = TfidfVectorizer(min_df= 3, stop_words="english", sublinear_tf=True, norm='l2', ngram_range=(1, 2))final_features = vectorizer.fit_transform(df['cleaned']).toarray()final_features.shape(3377, 2389)
so we ended up with 2389 features after converting the text into vectors. now we proceed with building our model and fit the model with our data. another important question that we should ask here is that do we really need to use all 2389 features? when dealing with very long vectors, sometimes it might be better to select your best features instead of using all of them. to do so, we use the SelectKbest method from SKlearn.feature_selection package. then we use the Chi2 score that can be used to select the n_features features with the highest values for the test chi-squared. “ chi-square test measures dependence between stochastic variables, so using this function “weeds out” the features that are the most likely to be independent of class and therefore irrelevant for classification.” quoted from sklearn documentation.
while building the model, we can choose from a wide range of classification algorithms. we tested four different algorithm: KNN, Multinomial Naive Bayes, Linear SVC, and Random Forrest. I investigated each classifier and came to the conclusion using common statistical measures (precision and recall) of classification performance to compare each and Random Forrest seemed to perform better than the rest. based on my own experience, Random Forrest does a better job when you have a multi-class problem. you can test other classification algorithm and let me know if you find some that provide better results.
now let’s put everything into a pipeline. here we are performing a sequence of transformations: first we transform the tfidf vectorizer, then we select K best features, and finally we use the classifier to fit the data. so we do all of them at once using a pipeline:
#first we split our dataset into testing and training set:# this block is to split the dataset into training and testing set X = df['cleaned']Y = df['class']X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.25)# instead of doing these steps one at a time, we can use a pipeline to complete them all at oncepipeline = Pipeline([('vect', vectorizer),                     ('chi',  SelectKBest(chi2, k=1200)),                     ('clf', RandomForestClassifier())])# fitting our model and save it in a pickle for later usemodel = pipeline.fit(X_train, y_train)with open('RandomForest.pickle', 'wb') as f:    pickle.dump(model, f)ytest = np.array(y_test)# confusion matrix and classification report(precision, recall, F1-score)print(classification_report(ytest, model.predict(X_test)))print(confusion_matrix(ytest, model.predict(X_test)))               precision    recall  f1-score   support          1       0.99      0.98      0.99       271          2       0.91      0.91      0.91        47          3       0.98      1.00      0.99       116          4       0.99      0.99      0.99       143          5       0.98      0.99      0.98       268avg / total       0.98      0.98      0.98       845Confusion Matrix:    1    2   3   4   51[[266   1   0   0   4] 2[  2  43   0   1   1] 3[  0   0 116   0   0] 4[  0   2   0 141   0] 5[  0   1   2   1 264]]
0.98 precision and recall ...!!! hmmm ... that as my first reaction. but then I looked over the confusion matrix and saw that there are not many missclassifications for each class and I think we are in a good place. we achieved high precision and recall without even doing any parameter tuning for the classification algorithm!
Now I really want to check if this is going to work for another dataset. moreover, I would like to evaluate this using other metrics like: AUC score and have a look at the ROC curve since the data is a bit imbalanced. two more things to do to have a more complete understanding: first is to use gridsearch to tune the parameters and see if it actually helps improving the results. second, it will be good to have maybe top 5 most important keywords for each class so later on we can use those keywords for future searching purposes. but this article is already long enough so I will answer to those questions in my next article. stay tuned!
your feedback and questions are highly appreciated.
The source code, python notebook, and datasets for this project are available on my github.