Analysing difference between Scrum & Agile using Pyspark
By: Kieran Tan Kah Wang
Tags: Data Science, Python, Agile, Scrum, Pyspark

Photo by Kelly Sikkema on Unsplash

Scrum and Agile are two modern software development methodologies, and most tech companies have been using these frameworks to develop or maintain a software application. So what exactly is the difference between these two? Just a Google search and you can probably see tons of articles comparing the differences between the two, but what if we try discovering the difference using a data science approach?

Dataset

So where do we start? People have been writing books about the two frameworks long before they are grabbing the attentions of tech companies and yeap, let’s analyse these books! Here’s the two books we will analyse:

Agile Processes in Software Engineering and Extreme Programming (Hubert Baumeister, Horst Lichter, Matthias Riebisch)

Scrum Handbook (Jeff Sutherland)

Initializing Pyspark

Pyspark is the Spark Python API that exposes the Spark programming model to Python. Sparkcontext is the main entry point for Spark functionality which will be created when we start a new application. This sparkcontext asks the master for some cores to do the processing, which will be set aside and won’t be used for other applications.

We will first create a SparkContext sc from pyspark that uses all available cores on the machine by setting master = local[*] . Then, we create a SparkSession object spark provided in the SparkContext object using the configuration set in SparkConf object.

# create entry points to spark

from pyspark import SparkContext, SparkConf # Spark

from pyspark.sql import SparkSession # Spark SQL # create or use existing spark context

sc = SparkContext.getOrCreate() # If there is no existing spark context, we now create a new context using all local processors

if (sc is None):

sc = SparkContext(master="local[*]", appName="Agile Scrum comparison")

spark = SparkSession(sparkContext=sc)

Creating Resilient Distributed Datasets (RDDs)

Resilient Distributed Datasets (RDDs) is a fundamental data structure of Spark which we will be creating two, one for each book on the cluster.

# read in first txt file

scrum_txt = sc.textFile('./Scrum Handbook.txt')

# print number of lines

print("Number of lines in Scrum Handbook is: ", str(scrum_txt.count())) # read in second txt file

agile_txt = sc.textFile('./Agile Processes in Software Engineering and Extreme Programming.txt')

# print number of lines

print("Number of lines in Agile Processes in Software Engineering and Extreme Programming is: ",\

str(agile_txt.count()))

We see that Agile Processes in Software Engineering and Extreme Programming has 21569 lines which is about 5 times the size of Scrum Handbook with 4617 lines.

Data cleaning

In this step, we perform the following text cleaning and manipulating steps:

split the lines into individual words (i.e. tokenization) using split with flatMap remove all characters which are not alphabets expect spaces using regular expression [A-Za-z\s] change all words into lowercase by using lower with map remove all leading or trailing spaces using strip with map

# import re for regular expression

import re # function to do the 3 text cleaning and manipulating steps

def cleantext(rdd):

# split the lines into individual words

words = rdd.flatMap(lambda x: x.split(' '))

# remove all characters which are not alphabets except spaces

alpha_words = words.filter(lambda x: re.findall('[A-Za-z\s]+',x))

# replace all characters which are not alphabets or spaces with ''

alpha_words_cleaned = alpha_words.map(lambda x: re.sub('[^A-Za-z\s]+','',x))

# change all to lower case

lower_alpha_words = alpha_words_cleaned.map(lambda x: x.lower())

# remove all leading or trailing spaces

cleaned_words = lower_alpha_words.map(lambda x: x.strip())

# return the result

return cleaned_words # apply cleantext function to scrum_txt

scrum_txt_cleaned = cleantext(scrum_txt) # apply cleantext function to agile_txt

agile_txt_cleaned = cleantext(agile_txt)

Counting the words

In this step, we perform a transformation step which first create a word pair for each word with a value of 1 using map and then sum the frequency of each word using reduceByKey . We then sort the words according to their frequency using sortBy with the options -1 to indicate descending order.

# function to transform the data and count the words

def count_words(rdd):

# transformation to convert each word to a key/value pair of (word, 1)

pairs = rdd.map(lambda x: (x,1))

# transformation to sum the frequency of each word

word_count = pairs.reduceByKey(lambda x,y: x+y)

# sort the words by frequency in descending order

sorted_word_count = word_count.sortBy(lambda x: -x[1])

# output

return sorted_word_count

We apply count_word to Scrum and display the top 20 most frequent words using take(20)

# apply count_words function to scrum_txt_cleaned and display top 20 most frequent words

scrum_txt_wordcount = count_words(scrum_txt_cleaned)

scrum_txt_wordcount.take(20)

We apply count_word to Agile and display the top 20 most frequent words using take(20)

# apply count_words function to agile_txt_cleaned and display top 20 most frequent words

agile_txt_wordcount = count_words(agile_txt_cleaned)

agile_txt_wordcount.take(20)

Comparing the top 20 words based on frequency in each book, they are mostly stopwords like the, of, and. which we should remove in the next step to better analyse the books.

Removing stopwords

In computing, stop words are words which are filtered out before or after processing of natural language data. In natural language processing, useless words (data), are referred to as stop words which provide little context in text analysis. We will get the stop words list from nltk.corpus package and use the function filter to remove those stop words in both books. Finally, we display the number of unique words in each book.

# import nltk package and stopwords

import nltk

from nltk.corpus import stopwords

# get the english stopwords since there are 16 languages available

eng_stopwords = set(stopwords.words('english')) # filter off those words in scrum_txt_wordcount that are in eng_stopwords list

scrum_txt_no_stopwords = scrum_txt_wordcount.filter(lambda x: x[0] not in eng_stopwords)

# get the number of unique words of each text after removal of stopwords

print("Number of unique non stopwords in Scrum Handbook is:",str(scrum_txt_no_stopwords.count()))

# filter off those words in agile_txt_wordcount that are in eng_stopwords list

agile_txt_no_stopwords = agile_txt_wordcount.filter(lambda x: x[0] not in eng_stopwords)

# get the number of unique words of each text after removal of stopwords

print("Number of unique non stopwords in Agile Processes in Software Engineering and Extreme Programming is:",\

str(agile_txt_no_stopwords.count()))

Average occurrence of a word

In this step, we find the average occurrence of a word by first finding the total number of words in each book using sum and map , and then dividing the value with the number of unique words.

# get total number of words in scrum_txt_no_stopwords

total_scrum_words = scrum_txt_no_stopwords.map(lambda x: x[1]).sum()

# find avg occurence of the words and return result in 3 decimal place

print("Average occurence of words in Scrum Handbook is:",\

round(total_scrum_words/scrum_txt_no_stopwords.count(),3))

# get total number of words in agile_txt_no_stopwords

total_agile_words = agile_txt_no_stopwords.map(lambda x: x[1]).sum()

# find avg occurence of the words and return result in 3 decimal place

print("Average occurence of words in Agile Processes in Software Engineering and Extreme Programming is:",\

round(total_agile_words/agile_txt_no_stopwords.count(),3))

Word distribution

We can analyse the distribution of words using visualisation matplotlib.pyplot by plotting the total number of words in each book followed by the average occurrence of word in each book.

# import the standard python plotting library and numpy for easy data manipulation

import matplotlib.pyplot as plt

import numpy as np

import math

# to show plot in jupyter notebook

%matplotlib inline # create three subplots

f, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize=(12,4), tight_layout=True) # Plot 1 - Total number of words

# prepare the data

y_axis_1 = [math.log10(total_scrum_words), math.log10(total_agile_words)]

x_axis = ('Scrum Handbook', 'Agile Processes')

bar_width = 0.5

# plot

ax1.bar(x_axis, y_axis_1, bar_width, align='center', color='C0')

ax1.set_xlabel('Book')

ax1.set_ylabel('Log Scale of number of words')

ax1.set_title('Total number of words in each book') # Plot 2 - Total number of unique words

# prepare the data

y_axis_2 = [scrum_txt_no_stopwords.count(), agile_txt_no_stopwords.count()]

x_axis = ('Scrum Handbook', 'Agile Processes')

bar_width = 0.5

# plot

ax2.bar(x_axis, y_axis_2, bar_width, align='center', color='C1')

ax2.set_xlabel('Book')

ax2.set_ylabel('Number of unique words')

ax2.set_title('Total number of unique words in each book') # Plot 3 -Average Occurence of word

# prepare the data

y_axis_3 = [total_scrum_words/scrum_txt_no_stopwords.count(), total_agile_words/agile_txt_no_stopwords.count()]

bar_width = 0.5

# plot

ax3.bar(x_axis, y_axis_3, bar_width, align='center', color='C2')

ax3.set_xlabel('Book')

ax3.set_ylabel('Average Occurence of word')

ax3.set_title('Average Occurence of word in each book') plt.show()

Image by author — matplotlib plots on words distribution

From the plots, it is clear that Agile Processes in Software Engineering and Extreme Programming is a longer book than Scrum Handbook with 1.25 times more words on the log10 scale, 3 times more unique words and each word in the former occurring twice more often than the latter, after removal of stopwords and text processing.

Most common words

Let’s now compare the top 15 most common words in each book.

# create two subplots

f, (ax1, ax2) = plt.subplots(1, 2, figsize=(12,4), tight_layout=True) # Plot 1 - Top 15 of scrum handbook

# prepare the data

top15_scrum_word = scrum_txt_no_stopwords.map(lambda x: x[0])

top15_scrum_word = top15_scrum_word.take(15)

top15_scrum_value = scrum_txt_no_stopwords.map(lambda x: x[1])

top15_scrum_value = top15_scrum_value.take(15)

# plot

bar_width = 0.5

ax1.bar(top15_scrum_word, top15_scrum_value, bar_width, align='center', color='C0',label='Scrum Handbook')

plt.setp(ax1.get_xticklabels(), rotation=90)

ax1.set_xlabel('Scrum Handbook')

ax1.set_ylabel('Count of words')

ax1.set_title('Total 15 words in Scrum Handbook')

ax1.legend() # Plot 2 - Top 15 of agile processes

# prepare the data

top15_agile_word = agile_txt_no_stopwords.map(lambda x: x[0])

top15_agile_word = top15_agile_word.take(15)

top15_agile_value = agile_txt_no_stopwords.map(lambda x: x[1])

top15_agile_value = top15_agile_value.take(15)

# plot

bar_width = 0.5

ax2.bar(top15_agile_word, top15_agile_value, bar_width, align='center', color='C1',label='Agile Processes')

plt.setp(ax2.get_xticklabels(), rotation=90)

ax2.set_xlabel('Agile Processes')

ax2.set_ylabel('Count of words')

ax2.set_title('Total 15 words in Agile Processes')

ax2.legend() plt.show()

Image by author — matplotlib plots on most common words

Comparing the top 15 of both books, the content of both books are quite similar as both revolves around product, team, development and software.

Summary

Based on this article I found on Google on the differences between Agile and Scrum, Agile is a project management philosophy which utilizes a core set of values or principles whereas Scrum is a specific Agile methodology that is used to facilitate a project.

Based on our analysis of the two books, we discovered a similar finding whereby Scrum Handbook focused on team development which is a way for project managers to facilitate projects and Agile Processes in Software Engineering and Extreme Programming more on software development in general.

There you go, hopefully this data analysis approach can give you another perspective in comparing differences rather than relying on our most reliable partner, Google!