Preprocessing Large Datasets: Online Retail Data with 500k+ Instances | by Dina Jankovic | Towards Data Science

Few months ago I had the opportunity to work on a project with a huge data set with over 500 000 rows! To be honest, it was the very first time I had to handle an insane amount of data, but I thought it would be fun to play with it and explore new data mining techniques. In this post, I am going to explain how I handled this issue and ended up with a clean data set that is convenient to work with. The R code is also provided.
The data set in question is available here at the UCI Machine Learning Repository. It is a transactional data set which contains all the transactions occurring between 01/12/2010 and 09/12/2011 for a UK-based and registered non-store online retail. The company mainly sells unique all-occasion gifts; many customers of the company are wholesalers. Attribute information can be found in the provided link. My ultimate goal was to do Market Basket Analysis on this data and figure out the association rules. However, the very first step is to clean the data.
I saved the data set as a .csv file and briefly looked over all the rows. I realized that I may not be able to notice the most important patterns and details because of the dimensionality issue. However, I noticed that some cells are empty, and those are missing values. They were recoded into NA for the sake of convenience. I also found where the missing values are, and how many.
dataset = read.csv("OnlineRetail.csv", na.strings = c("","NA"))attach(dataset)#checking if there any missing values, where are they missing, and how many of them are missingany(is.na(dataset))[1] TRUEapply(dataset, 2, function(x) any(is.na(x)))InvoiceNo StockCode Description Quantity InvoiceDate UnitPrice CustomerID CountryFALSE FALSE TRUE FALSE FALSE FALSE TRUE FALSEsum(is.na(CustomerID))[1] 135080sum(is.na(Description))[1] 1454
InvoiceNo is of type integer; however, it’d better to be of type character so that we can apply string functions. The leading and trailing white spaces were removed with the trim function both in InvoiceNo and Description.
dataset$InvoiceNo = as.character(InvoiceNo)trim = function (x) gsub("^\\s+|\\s+$", "", x)dataset$InvoiceNo = trim(InvoiceNo)dataset$Description = trim(as.character(Description))
The invoice numbers starting with C are actually cancellations, so we may want to get rid of them. I introduced a function called is_C which checks if a string starts with C. We subset the original data, and obtain dataset2 (without the cancellations) and dataset3 (without empty item descriptions). We can see that the original data set had 541 909 observations, and now we have 531 167 observations in dataset3. There were nearly 10 000 useless observations that we just got rid of!
is_C = function (x) startsWith(x,"C")dataset2 = dataset[which(!is_C(dataset$InvoiceNo)),] #subsettingdataset3 = subset(dataset2,!is.na(dataset2$Description)) #subsetting
Next, I looked over the rows one more time, and figured out that many item descriptions don’t make any sense. For example, I saw symbols such as “?” and words like DAMAGED, SMASHED, LOST, etc. It’d be a good idea to get rid of these undesirable items. I came up with a list of interesting buzzwords:
buzzwords = c("WRONG","LOST", "CRUSHED", "SMASHED", "DAMAGED", "FOUND", "THROWN", "MISSING", "AWAY", "\\?", "CHECK", "POSTAGE", "MANUAL", "CHARGES", "AMAZON", "FEE", "FAULT", "SALES", "ADJUST", "COUNTED", "LABEL", "INCORRECT", "SOLD", "BROKEN", "BARCODE", "CRACKED", "RETURNED", "MAILOUT", "DELIVERY", "MIX UP", "MOULDY", "PUT ASIDE", "ERROR", "DESTROYED", "RUSTY")
If any of the items contain one of these words, it will be removed. The following function isUndesirable tests whether or not an item is “undesirable”.
library(stringr)   #function str_detectisUndesirable = function(x){c = FALSE   #assume that the string is undesirable (FALSE), and perhaps switch to desirable (TRUE)for (i in 1:(length(buzzwords))){  c = c || ifelse(str_detect(toupper(x),buzzwords[i]),TRUE,FALSE)}#now we know whether or not the string is undesirablereturn(c)}
Now we can subset dataset3 such that Quantity is greater than 0 (otherwise it doesn’t make sense), and the item name is “desirable”. We obtain dataset4, and finally dataset5. The InvoiceDate was converted to a POSIXct object, and the exact time of order was excluded. The Description column was factored.
dataset4 = subset(dataset3, dataset3$Quantity > 0)dataset5 = dataset4[which(!isUndesirable2(as.character(dataset4$Description))),]Time = format(as.POSIXct(strptime(dataset5$InvoiceDate,"%Y-%m-%d %H:%M",tz="")) ,format = "%H:%M:%S")dataset5$InvoiceDate = as.Date(dataset5$InvoiceDate)dataset5$Description = as.factor(dataset5$Description)
And that’s it! Now we have dataset5 with 530 653 observations, ready to be used for Market Basket Analysis. Note that this data set still has missing values, in the CustomerID column. However, that is irrelevant — why would we care about a customer’s ID? The invoice number is more important for finding the association rules and further analysis. However, if we wanted to use this data in another analysis, we could preprocess it in a different way.
For the full R code, please visit my GitHub profile.