Make an Impact Using Data Analysis and Visualization Using Python, Spacy and Tableau
By: Laxman Singh
Tags: Technology, Learning, Python, Data Science, Data Visualization

Data Cleaning and Transformation

Data set was huge. Data analysis done by picking the data in chunks. Remove all the questions with delta answers negative or zero. Here, Delta answers are calculated by difference between time questions asked and respond to that question. Text is cleansed by removing punctuations and unwanted symbols. Two corpus created one with stopwords and another without stopwords for analysis. Existing python library spacy is used in place of NLTK for data cleansing and text processing. It is very fast and has many more existing useful features than NLTK

Next Steps to Explore More

Data Normalization Step : Normalization of data will help to remove outliers, missing value. In simple, in case data is skewed, normalization will help to make data better for model to perform better. And, as expected, there were huge differences found after and before normalization. And, text became more manageable after normalization.

Score Analysis Step : Score is the upvote given to question or answer. Analysis of score here means count the score and as per analysis, score always lies between 0 and 100.

Question Body Analsis Step : Question body has many insights and findings. As per analysis, length of question results in quick response or not. And, it is also related to upvote too.

Text Summarization Step : Simple technique applied to summarize the text.

Converted paragraph to sentences

Removed special characters or stop words

Tokenized and find weighted frequency of occurrence (TFID or PMI)

Replace words by weighted frequency in original sentence

Below are some of the graphs to analyse more about data.

Modeling and Sentiment Analysis

● Data Ingestion, preprocessing, transformation, clustering, train and test and predict time series.

● Visualization and decision making

Sentiment analysis can be performed using Spacy, afinn and TextBlob. Scores were already there in the data but sentiment analysis helps to give us more insights of data. In simple, get the given sentiments and sum up with sentiments provided using our own trained model. And, it is assumed it will help to give more conclusive sentiment out of responses.

Spacy is library in python which has all the necessary libraries to perform NLP tasks including NER and derive meaning from data. It has pre-trained langualge models for small, medium and large which can be utilized based on problem statement and requirements.

Final Output and Dashboard for Visualization

Dashboard is developed based on the files generated after sentiment analysis and normalization of the data performed using python. Final data frames are stored as csv file for using in Tableau. Tried developing around 12 different views in the dashboard to have good prospects of the problem and have a nicer conclusion. More about dashboard

Interactive Dashboard to Visualize and Understand Data set

Dashboard is there to visualize Total records, total answers, Q&A, Q&A ratio, number of answers count to questions, top 10 technologies discussed, technologies with positive comments, tag counts, tag analysis for filtering dashboard dynamically, and Q&A analysis. This gives good visualization view to understand and interprete data at one glance

Word cloud of different technologies and most used words to give analysis of the data to feed in for modeling and sentiment analysis. It is very good view as you can see whether wrong words are not getting picked up for analysis. If yes, include them in stop-words removal step.

d

Conclusion

Goal in this analysis is to explore the reasoning of community process in question answering at stack overflow. This analysis will be helpful to different technologists to know the trending technologies and technologies those are downgrading or upgrading their popularity. This also helps to determine technology roadmap and journey for future. Also, it could be help to plan resources and capabilities of the organization.

This analysis can be used to determine market demand of lacking skills and which product got lot of bug and issues. So, it will help to a company architect or strategic person in deciding product they can select for enterprise standard and build capabilities in that area.

There is a growing demand in this domain which can be analyzed, and this can be used for informed or data based decision making. This can be taken to the next level to voice-based response using AI powered devices such as SIRI, Amazon’s Alexa, Google Assistant etc., interfacing with knowledge base. Also chatbot can leverage the auto answering or solution capability using the knowledge base. Overall, this analysis could help every kind of technologists to derive information useful to them. Happy Coding!