Industrial Classification of Websites by Machine Learning with hands-on Python | by Ridham Dave | Towards Data Science

Hey folks, welcome to my first technical tutorial. In this tutorial, I would like to explain extraction, cleaning and classification of websites into different categories. I will use python environment to run my code for data scraping and use neural network to classify websites.
Text classification is one of the widely used natural language processing(NLP) task in a lot of different areas of Data Science. An efficient text classifier can automatically distinguish the data into categories efficiently with the use NLP algorithms.
Text Classification is an example of supervised machine learning task since a labelled dataset containing text documents and their labels is used for train a classifier.
Some common techniques for text classification are :
Naive Bayes ClassifierLinear ClassifierSupport Vector MachineBagging ModelsBoosting ModelsDeep Neural Networks
Naive Bayes Classifier
Linear Classifier
Support Vector Machine
Bagging Models
Boosting Models
Deep Neural Networks
Web scraping, web harvesting, or web data extraction is data scraping used for extracting data from websites. Generally, this is done with software that simulates human Web surfing to collect specified bits of information from different websites.
Some techniques that can be used for web scraping are :
Human copy-and-pasteText pattern matchingHTTP programmingHTML parsingDOM parsingVertical aggregationSemantic annotation recognizingComputer vision web-page analysis
Human copy-and-paste
Text pattern matching
HTTP programming
HTML parsing
DOM parsing
Vertical aggregation
Semantic annotation recognizing
Computer vision web-page analysis
In this tutorial, we will try to implement the complete model into three different modules :
Data ScrapingClassification based on keywords for creating a training data setApplying neural network for actual testing model
Data Scraping
Classification based on keywords for creating a training data set
Applying neural network for actual testing model
In this module, I will use Python 3.5 environment to implement my scripts. So follow along for complete reference.
To extract web data many different packages are available, but in this tutorial I will use requests.
import requestsurl='https://medium.com/'try:    page = requests.get(url)        #to extract page from website    html_code = page.content        #to extract html code from pageexcept Exception as e:    print(e)
In the above code, the requests.get() method will request the page from website using https protocol and load the page into the object “page”. The next line of code will move the HTML code to String html_code. So until now we have extracted the data from websites but it is still in HTML format which is far different than the actual text.
To extract complete text data from the HTML page, we have two highly preferred packages , BeautifulSoup and html2text. Using the html_code string found from the previous step, we can apply any of the following two methods.
from bs4 import BeautifulSouptry:    soup = BeautifulSoup(html_code, 'html.parser')  #Parse html code    texts = soup.findAll(text=True)                 #find all text    text_from_html = ' '.join(texts)                   #join all textexcept Exception as e:    print(e)
In the above snippet, BeautifulSoup package will parse the HTML code and assign the data to soup object. The findall() function finds all visible text from code and return a list of Strings which we store in texts. And finally we join all individual text into a common string using join() function.
import html2texth = html2text.HTML2Text()                 #Initializing objecth.ignore_links = True                     #Giving attributes try:    text = h.handle(html_code)            #handling the HTML code     text_from_html=text.replace("\n"," ") #replacing next line charexcept Exception as e:    print(e)
In this alternate block, we use package html2text to parse the string and directly get the text from the HTML code. Also we need to replace blank lines with spaces and finally find text_from_html.
Similarly we can use a loop on some 1000+ urls and extract data from those sites as well and store them in a csv(Comma Separated File) format, which we can further use in the classification module.
For any machine learning algorithm, we need some training set and test set for training the model and testing the accuracy of that model. Hence to create the set of data for the model, we already have the text from different websites, we will just classify them according to the keywords, and then apply the results in the next module.
In this tutorial, we are going to classify websites into three categories namely:
Technology, Office, & Education products website(Class_1)Consumer products website(Class_2)Industrial Tools and Hardware products website(Class_3)
Technology, Office, & Education products website(Class_1)
Consumer products website(Class_2)
Industrial Tools and Hardware products website(Class_3)
The approach here is that we will have certain keywords belonging to the particular category, and we will match those keywords with the text and find the class with the maximum Matching_value.
Matching_value = (Number of keywords matched with one industry)/(Total number of keywords matched)
So accordingly we have a list of keywords for the individual categories as follows:
Class_1_keywords = ['Office', 'School', 'phone', 'Technology', 'Electronics', 'Cell', 'Business', 'Education', 'Classroom']Class_1_keywords = ['Restaurant', 'Hospitality', 'Tub', 'Drain', 'Pool', 'Filtration', 'Floor', 'Restroom', 'Consumer', 'Care', 'Bags', 'Disposables']Class_3_keywords = ['Pull', 'Lifts', 'Pneumatic', 'Emergency', 'Finishing', 'Hydraulic', 'Lockout', 'Towers', 'Drywall', 'Tools', 'Packaging', 'Measure', 'Tag ']keywords=Class_1_keywords + Class_2_keywords + Class_3_keywords
Now, we will use KeywordProcessor to find keywords inside the text recieved from the the urls.
KeywordProcessor is available in flashtext package on pypi.
from flashtext.keyword import KeywordProcessorkp0=KeywordProcessor()for word in keywords:    kp0.add_keyword(word)kp1=KeywordProcessor()for word in Class_1_keywords:    kp1.add_keyword(word)kp2=KeywordProcessor()for word in Class_2_keywords:    kp2.add_keyword(word)kp3=KeywordProcessor()for word in Class_3_keywords:    kp3.add_keyword(word)
In the above code, we will load KeywordProcessor objects with the keywords which we will use further for finding the matching keywords.
To find percentage value of Matching_value we define a function percentage as follows:
def percentage1(dum0,dumx):    try:        ans=float(dumx)/float(dum0)        ans=ans*100    except:        return 0    else:        return ans
We will now use extract_keywords(string) method to find keywords present in the text. And we will find length of that list to find the number of matching keywords in the text. The following function will find the percentage and the class with maximum percentage is selected.
def find_class:    x=str(text_from_html)    y0 = len(kp0.extract_keywords(x))    y1 = len(kp1.extract_keywords(x))    y2 = len(kp2.extract_keywords(x))    y3 = len(kp3.extract_keywords(x))    Total_matches=y0       per1 = float(percentage1(y0,y1))    per2 = float(percentage1(y0,y2))    per3 = float(percentage1(y0,y3))    if y0==0:        Category='None'    else:        if per1>=per2 and per1>=per3:            Category='Class_1'        elif per2>=per3 and per2>=per1:            Category='Class_2'        elif per3>=per1 and per3>=per2:            Category='Class_3'    return Category
Using a loop on the above function, we can find the category of all websites based on the keywords. We will save the classified data into a file Data.csv which we will use further. So now we have our Data Set ready for applying neural network for classification.
In the following implementation, we will create a neural network from scratch and will use NLTK word tokenizer for preprocessing. First we need to import our dataset obtained from the above steps and load it into an list.
import pandas as pddata=pd.read_csv('Data.csv')data = data[pd.notnull(data['tokenized_source'])]data=data[data.Category != 'None']
The above code will load and clean the classified data. The NULL values will be removed.
The following code will create a dictionary of DATA against its class.
for index,row in data.iterrows():    train_data.append({"class":row["Category"], "sentence":row["text"]})
For applying neural network, we need to convert the Language Words into mathematical notations, which will be used for calculations. We will form a list of all the words across all the strings.
words = []classes = []documents = []ignore_words = ['?']# loop through each sentence in our training datafor pattern in training_data:    # tokenize each word in the sentence    w = nltk.word_tokenize(pattern['sentence'])    # add to our words list    words.extend(w)    # add to documents in our corpus    documents.append((w, pattern['class']))    # add to our classes list    if pattern['class'] not in classes:        classes.append(pattern['class'])# stem and lower each word and remove duplicateswords = [stemmer.stem(w.lower()) for w in words if w not in ignore_words]words = list(set(words))# remove duplicatesclasses = list(set(classes))print (len(documents), "documents")print (len(classes), "classes", classes)print (len(words), "unique stemmed words", words)
For example, the output will be :
1594 documents3 classes ['Class_1', 'Class_3', 'Class_2']unique stemmed words 40000
Now, we will create a list of tokenized words for the pattern and also create a bag of words by using NLTK Lancaster Stemmer.
from nltk.stem.lancaster import LancasterStemmerstemmer = LancasterStemmer()# create our training datatraining = []output = []# create an empty array for our outputoutput_empty = [0] * len(classes)# training set, bag of words for each sentencefor doc in documents:    # initialize our bag of words    bag = []    # list of tokenized words for the pattern    pattern_words = doc[0]    # stem each word    pattern_words = [stemmer.stem(word.lower()) for word in pattern_words]    # create our bag of words array    for w in words:        bag.append(1) if w in pattern_words else bag.append(0)    training.append(bag)    # output is a '0' for each tag and '1' for current tag    output_row = list(output_empty)    output_row[classes.index(doc[1])] = 1    output.append(output_row)print ("# words", len(words))print ("# classes", len(classes))
Output:
# words 41468# classes 3
Now, we do the final preprocessing on the data and create some functions.
Sigmoid Function
def sigmoid(x):    output = 1/(1+np.exp(-x))    return output# convert output of sigmoid function to its derivativedef sigmoid_output_to_derivative(output):    return output*(1-output)
Cleaning function
def clean_up_sentence(sentence):    # tokenize the pattern    sentence_words = nltk.word_tokenize(sentence)    # stem each word    sentence_words = [stemmer.stem(word.lower()) for word in sentence_words]    return sentence_words
Bag Of Words function
def bow(sentence, words, show_details=False):    # tokenize the pattern    sentence_words = clean_up_sentence(sentence)    # bag of words    bag = [0]*len(words)      for s in sentence_words:        for i,w in enumerate(words):            if w == s:                 bag[i] = 1                if show_details:                    print ("found in bag: %s" % w)    return(np.array(bag))
The final function which will be used in neural network : Think function
def think(sentence, show_details=False):    x = bow(sentence.lower(), words, show_details)    if show_details:        print ("sentence:", sentence, "\n bow:", x)    # input layer is our bag of words    l0 = x    # matrix multiplication of input and hidden layer    l1 = sigmoid(np.dot(l0, synapse_0))    # output layer    l2 = sigmoid(np.dot(l1, synapse_1))    return l2
Now we are all set to train our model of Neural Network. We are going to implement it through scratch and will be using Logistic regression into each neuron. With just one layer, but with 50000 epochs, we will be training our model. The complete training example will run on CPU.
def train(X, y, hidden_neurons=10, alpha=1, epochs=50000, dropout=False, dropout_percent=0.5):    print ("Training with %s neurons, alpha:%s, dropout:%s %s" % (hidden_neurons, str(alpha), dropout, dropout_percent if dropout else '') )    print ("Input matrix: %sx%s    Output matrix: %sx%s" % (len(X),len(X[0]),1, len(classes)) )    np.random.seed(1)    last_mean_error = 1    # randomly initialize our weights with mean 0    synapse_0 = 2*np.random.random((len(X[0]), hidden_neurons)) - 1    synapse_1 = 2*np.random.random((hidden_neurons, len(classes))) - 1    prev_synapse_0_weight_update = np.zeros_like(synapse_0)    prev_synapse_1_weight_update = np.zeros_like(synapse_1)    synapse_0_direction_count = np.zeros_like(synapse_0)    synapse_1_direction_count = np.zeros_like(synapse_1)            for j in iter(range(epochs+1)):        # Feed forward through layers 0, 1, and 2        layer_0 = X        layer_1 = sigmoid(np.dot(layer_0, synapse_0))                        if(dropout):            layer_1 *= np.random.binomial([np.ones((len(X),hidden_neurons))],1-dropout_percent)[0] * (1.0/(1-dropout_percent))        layer_2 = sigmoid(np.dot(layer_1, synapse_1))        # how much did we miss the target value?        layer_2_error = y - layer_2        if (j% 10000) == 0 and j > 5000:            # if this 10k iteration's error is greater than the last iteration, break out            if np.mean(np.abs(layer_2_error)) < last_mean_error:                print ("delta after "+str(j)+" iterations:" + str(np.mean(np.abs(layer_2_error))) )                last_mean_error = np.mean(np.abs(layer_2_error))            else:                print ("break:", np.mean(np.abs(layer_2_error)), ">", last_mean_error )                break                        # in what direction is the target value?        # were we really sure? if so, don't change too much.        layer_2_delta = layer_2_error * sigmoid_output_to_derivative(layer_2)        # how much did each l1 value contribute to the l2 error (according to the weights)?        layer_1_error = layer_2_delta.dot(synapse_1.T)        # in what direction is the target l1?        # were we really sure? if so, don't change too much.        layer_1_delta = layer_1_error * sigmoid_output_to_derivative(layer_1)                synapse_1_weight_update = (layer_1.T.dot(layer_2_delta))        synapse_0_weight_update = (layer_0.T.dot(layer_1_delta))                if(j > 0):            synapse_0_direction_count += np.abs(((synapse_0_weight_update > 0)+0) - ((prev_synapse_0_weight_update > 0) + 0))            synapse_1_direction_count += np.abs(((synapse_1_weight_update > 0)+0) - ((prev_synapse_1_weight_update > 0) + 0))                        synapse_1 += alpha * synapse_1_weight_update        synapse_0 += alpha * synapse_0_weight_update                prev_synapse_0_weight_update = synapse_0_weight_update        prev_synapse_1_weight_update = synapse_1_weight_update    now = datetime.datetime.now()    # persist synapses    synapse = {'synapse0': synapse_0.tolist(), 'synapse1': synapse_1.tolist(),               'datetime': now.strftime("%Y-%m-%d %H:%M"),               'words': words,               'classes': classes              }    synapse_file = "synapses.json"    with open(folder_path+synapse_file, 'w') as outfile:        json.dump(synapse, outfile, indent=4, sort_keys=True)    print ("saved synapses to:", synapse_file)
And finally we will train the model :
import timeX = np.array(training)y = np.array(output)start_time = time.time()train(X, y, hidden_neurons=10, alpha=0.1, epochs=50000, dropout=False, dropout_percent=0.2)elapsed_time = time.time() - start_timeprint ("processing time:", elapsed_time, "seconds")
Output :
Training with 10 neurons, alpha:0.1, dropout:False Input matrix: 1594x41468 Output matrix: 1x3delta after 10000 iterations:0.0665105275385delta after 20000 iterations:0.0610711168863delta after 30000 iterations:0.0561908365355delta after 40000 iterations:0.0533465919346delta after 50000 iterations:0.0461560407785saved synapses to: synapses.jsonprocessing time: 33060.51151227951 seconds
As we can see, it took almost 11 hours hours to train the model. And after such an intensive calculations, we are ready to test the data.
The function to test the data :
# probability thresholdERROR_THRESHOLD = 0.2# load our calculated synapse valuessynapse_file = 'synapses.json' with open(synapse_file) as data_file:     synapse = json.load(data_file)     synapse_0 = np.asarray(synapse['synapse0'])     synapse_1 = np.asarray(synapse['synapse1'])def classify(sentence, show_details=False):    results = think(sentence, show_details)    results = [[i,r] for i,r in enumerate(results) if r>ERROR_THRESHOLD ]     results.sort(key=lambda x: x[1], reverse=True)     return_results =[[classes[r[0]],r[1]] for r in results]    #print ("\n classification: %s" % ( return_results))    return return_results
Lets test the model with its accuracy :
classify("Switchboards Help KA36200 About Us JavaScript seems to be disabled in your browser You must have JavaScript enabled in your browser to utilize the functionality of this website Help Shopping Cart 0 00 You have no items in your shopping cart My Account My Wishlist My Cart My Quote Log In BD Electrical Worldwide Supply Remanufacturing the past SUSTAINING THE FUTURE Hours and Location Michigan Howell")
Output:
[['Class_3', 0.97663437888614435]]
classify("  New Website Testimonial Policies Parts Catalog Contact Support Forum Documentation Themes WordPress Blog Products Spindle Parts Latest News Kennard Parts Suggest Ideas Legal/Disclaimers WordPress Planet News About CDT Home Latest News Testimonial Products Parts Catalog About CDT History Staff Policies Centrum Legal Disclaimers Contact About CDT Custom Drilling Technologies established in 1990 has been providing superior customer service to the printed circuit board industry for almost 20 years We specialize in Excellon Drilling and Routing Equipment Parts and Service Our staff has over sixty years of combined experience in the design building troubleshooting operation programming")
Output:
[['Class_1', 0.9620297535870017]]
As you can see, we are getting pretty high accuracy on these tests. I have tried this model on different data, and have found pretty higher accuracy.
An accuracy of around 95%+ is considered pretty accurate in this type of model with just one layer. For further classification by different models, we can use Keras or Tensorflow. To decrease the time to train the model, we can use a NVIDIA GPU.
And now we can easily scrape data and classify its category, with the help of Deep Neural Network using Back Propogation .
In the further tutorials, I will try to explain Keras and Tensorflow’s working and hands-on.
Please share your feedback about this tutorial in comment section below or via my LinkedIn page: https://www.linkedin.com/in/ridhamdave/ . Also share your doubts regarding this tutorial.