Building & Deploying a Vertical Search Engine from Scratch
Tags: Ranking Algorithm, Search Ranking, Search Engines, Flask, Python

Data Collection & Cleaning

Because this search engine was built from scratch, the first step of this project was gathering the text documents related to the spaceflight programs for users to search through. To do this, I scraped multiple websites like Wikipedia.com and nasa.com for information relevant to the Shuttle and/or Apollo space programs. Many of the sites were structured in a way that made scraping & crawling do-able, but time-consuming nevertheless. I used the BeautifulSoup package in Python to collect the data on these websites. The BS4 package is widely used and there are tons of helpful tutorials out there to help you get started using it. The documentation is also really helpful and is a good place to start.

After I had collected all of the documents, I ended up with 1820 files that were pretty highly skewed in terms of length:

Min doc length: 14

Max doc length: 13743

Average doc length: 852.6697802197803

Since the length of documents was so skewed and the range was so wide I did some filtering. I dropped all of the documents that were under 30 tokens under the assumption that they likely didn’t contain much information. I also limited the max document length to 1000 tokens based on the belief that most of the useful info in these long documents is towards the beginning. These steps helped make the documents a little more similar to each other which will be helpful as we implement different ranking algorithms:

Min filtered doc length: 30

Max filtered doc length: 1000

Avgerage filtered doc length: 559.2797956867196

Ranking Algorithms

Once I had all of my documents collected, the next step was implementing different ranking algorithms to see which one performed the best. I tried a total of 8 different approaches in total. I focused on different term-weighting approaches for this project. These types of systems rank documents based on scoring derived from things like term frequencies in documents, document length, and term frequencies in queries. Using these types of scoring/ranking formulas determine document relevance with respect to the user’s query by determining the importance of words present across documents and queries.

I first established a baseline by simply returning random documents to a given query. While it is no surprise that this approach gave extremely poor results, it was useful because it showed which models gave the biggest boost in performance relative to simply randomly returning documents.

The next approach I took was ranking documents based solely on term frequencies. In this naive, bag-of-words approach, document scores were assigned based on how many of the query terms were present in the document. For example, if my query is “moon landing”, the score of each document will be the summation of how many times the words “moon” and “landing” occur in that document.

I also tried ranking documents based on their TF-IDF weights. In this simple approach, I scored documents by summing the TF-IDF weight for each query term. I found this article helpful as I was building everything for this approach from scratch. There is also a TF-IDF implementation in scikit learn that is quite popular.

TF-IDF:

Where TF:

And IDF:

The next approaches I tried were BM25, BM25L, and BM25+. I used the Rank-BM25 package in Python to implement all three of these algorithms.

BM25:

BM25L:

BM25+:

Whilst researching different ranking algorithms to implement I came across the paper “Analysing Ranking Functions in Information Retrieval Using Constraints” where the authors Cummins and O’Riordan explored different term weighting functions for information retrieval. I used two of the approaches outlined in their paper. I implemented the axiomatic term-weighting scheme as well as the divergence from randomness scheme that was originally proposed by Amati and van Rijsbergen in their paper “Probabilistic Models of Information Retrieval Based on Measuring the Divergence from Randomness”

Axiomatic Term-Weighting:

Divergence from Randomness:

Testing & Evaluation

To evaluate the effectiveness of the various schemes, I evaluated the precision and recall @ 10 for a total of 20 different test queries.

query_dict = {

‘query1’: ‘approach and landing tests’,

‘query2’: ‘first space shuttle launch’,

‘query3’: ‘sally ride’,

‘query4’: ‘STS-26’,

‘query5’: ‘last space shuttle launch’,

‘query6’: ‘untethered space walk’,

‘query7’: ‘first african american astronaut’,

‘query8’: ‘space shuttle challenger’,

‘query9’: ‘space shuttle columbia’,

‘query10’: ‘oldest astronaut to launch in shuttle’,

‘query11’: ‘what is the space shuttle program’,

‘query12’: ‘atlantis first flight’,

‘query13’: ‘first space walk’,

‘query14’: ‘STS-51-j’,

‘query15’: ‘STS-127’,

‘query16’: ‘STS-124’,

‘query17’: ‘apollo 1 fire’,

‘query18’: ‘first moon landing’,

‘query19’: ‘what is the hubble space telescope’,

‘query20’: ‘international space station’

}

For each of these test queries, I annotated the documents based on which were relevant to each query so that I would be able to measure how good each approach was at returning useful documents. I used the precision and recall implementations in scikit learn to calculate the scores.

Below you can see the average precision and recall results across each test query for each different ranking approach:

From the results, you can see that the axiomatic and divergence from randomness approaches achieved the highest precision scores topping out at close to 0.25. Similarly, these two approaches are among the top in terms of recall along with most of the other approaches as well. Each of the different schemes offered a decent improvement in precision and a modest improvement in recall.

I ultimately decided on implementing the divergence of randomness ranking scheme for my final product. I chose this method because of the strong performance in both precision and recall, and because for me, it was slightly faster to compute than the axiomatic approach.

Deploying the Finished Product

Now that I had determined the best performing ranking algorithm out of the approaches that I considered, I wanted to make deploy it so other people could see how the search engine performs for themselves. I decided to build a web app using Flask and deploy it on PythonAnywhere.

Check out the finished product here!

While it’s not the main focus of this post, there are some really good tutorials for Flask here and here that I found helpful, along with the documentation. As far as PythonAnywhere goes, it’s more simple than you’d think to get your deployed app up and running — especially if you follow this video tutorial and this write-up.

Reflection & Conclusion

Ultimately, I would consider this project a success! Although the precision and recall scores weren’t particularly impressive, some of the results were promising and it’s always fun to successfully implement some cool algorithms from the literature. I’d be interested in implementing some more sophisticated ranking approaches — namely some of the learning-to-rank and other neural methods — to see how they perform against the term-weighting schemes that I mainly focused on. Also, I’d love to have a bunch more data and more annotations with test queries! I would also be intrigued to implement some kind of query expansion to see if it could potentially improve performance at all.

Check out the GitHub repo with all the code here!

Thanks for reading!