Create a full search engine via Flask, ElasticSearch, javascript, D3js, asynchrous request (xml http request) and Bootstrap | by Adrien Sieg | Towards Data Science

A search engine is a system — displaying some filters — in order to customize your search results and allows you to find exactly what you want. When a user queries a search engine, relevant results are returned based on the search engine’s algorithm.
Filters turn out to a list (dropdown list), a table, ... any elements allowing to broaden or tighten a search in order to get relevant results (results you are actually interested in).
Google is the reigning king of ‘spartan searching’, and is the single most used search engine in the world. Google offers a full free text search filter. A user enters keywords or key phrases into a search engine and receives a personal and relevant result. In this article, we are going to pass Google.
Let’s consider a dataset coming from WineEnthusiast when it comes to wine reviews. The code for the scraper can be found here. The search engine still won’t be able to taste the wine, but theoretically, it could retrieve the wine based on a description that a sommelier could give.
Country: The country that the wine comes from
Description: A few sentences from a sommelier describing the wine’s taste, smell, look, feel, etc.
Designation: The vineyard within the winery where the grapes that made the wine are from
Points: The number of points WineEnthusiast rated the wine on a scale of 1–100 (though they say they only post reviews for wines that score >=80)
Price: The cost for a bottle of the wine
Province: The province or state that the wine is from
Variety: The type of grapes used to make the wine (ie Pinot Noir)
Winery: The winery that made the wine
To kick off the journey, we will focus on three specific countries and three regions of wines.
It works well if you know that Bordeaux or Champagne are regions within France, that Lombardy and Tuscany are regions of Italy and finally that Andalucia and Catalunia are regions of Spain. If you don’t, you are stuck and get nothing. That’s a huge problem because you have to try all possibilities to luckily get something. How to deal with this issue? By implementing a parent/children logic within filters.
When it comes to app itself — there is a back-end (app.py) and a front-end (index.html). When you select an element within dropdown field, an XMLHttpRequest is sent to the server, and results are returned (from the server). The XMLHttpRequest object can be used to request data from a web server.
App.py
The XMLHttpRequest object is a developers dream, because you can:
Update a web page without reloading the page
Request data from a server — after the page has loaded
Receive data from a server — after the page has loaded
Send data to a server — in the background
The onreadystatechange property specifies a function to be executed every time the status of the XMLHttpRequest object changes:
Templates/index.html
This is not very pleasant to have all options possible when a given dropdown list depends on another one. If you click on France and then Catalunia, we are stuck and get nothing because Catalunia is not in France... Too bad. Either you try all possibilities to get something or we implement parent-children properties as below:
Parent-children properties:
Connect ElasticSearch database to python. Elasticsearch is a search engine based on Lucene library. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents. Elasticsearch has quickly become the most popular search engine, and is commonly used for log analytics, full-text search, security intelligence, business analytics, and operational intelligence use cases.
from time import timefrom elasticsearch import Elasticsearchfrom elasticsearch.helpers import bulkimport pandas as pdimport requestsres = requests.get(‘http://localhost:9200')print(res.content)#connect to our clusterfrom elasticsearch import Elasticsearches = Elasticsearch([{‘host’: ‘localhost’, ‘port’: 9200}])
Transform a csv file into json format to put it into ElasticSearch.
def index_data(data_path, index_name, doc_type):   import json   f = open(data_path)   csvfile = pd.read_csv(f, iterator=True, encoding=”utf8")    r = requests.get(‘http://localhost:9200')   for i,df in enumerate(csvfile):        records=df.where(pd.notnull(df), None).T.to_dict()       list_records=[records[it] for it in records]       try :          for j, i in enumerate(list_records):              es.index(index=index_name, doc_type=doc_type, id=j, body=i)        except :           print(‘error to index data’)
Useful to access the desired page directly without having to navigate from the home page. The route() decorator is used to bind URL to a function.
@app.route(‘/hello’)def hello_world():   return ‘hello world’
URL ‘/hello’ rule is bound to the hello_world() function. As a result, if a user visits http://localhost:5000/hello URL, the output of the hello_world() function will be rendered in the browser.
The add_url_rule() function of an application object is also available to bind a URL with a function as below:
def hello_world():   return ‘hello world’app.add_url_rule(‘/’, ‘hello’, hello_world)
Http protocol is the foundation of data communication in world wide web.
Templates/index.html
<html>   <body>      <form action = "http://localhost:5000/login" method = "post">         <p>Enter Name:</p>         <p><input type = "text" name = "nm" /></p>         <p><input type = "submit" value = "submit" /></p>      </form>         </body></html>
app.py
from flask import Flask, redirect, url_for, requestapp = Flask(__name__)@app.route('/success/<name>')def success(name):   return 'welcome %s' % name@app.route('/login',methods = ['POST', 'GET'])def login():   if request.method == 'POST':      user = request.form['nm']      return redirect(url_for('success',name = user))   else:      user = request.args.get('nm')      return redirect(url_for('success',name = user))if __name__ == '__main__':   app.run(debug = True)
It is possible to return the output of a function bound to a certain URL in the form of HTML.
hello() function will render ‘Hello World’ with <h1> tag attached to it.
from flask import Flaskapp = Flask(__name__)@app.route('/')def index():   return '<html><body><h1>'Hello World'</h1></body></html>'if __name__ == '__main__':   app.run(debug = True)
Generating HTML content from Python code is cumbersome, especially when variable data and Python language elements like conditionals or loops need to be put. This would require frequent escaping from HTML.
This is where one can take advantage of Jinja2 template engine, on which Flask is based. Instead of returning hardcode HTML from the function, a HTML file can be rendered by the render_template() function.
from flask import Flask, render_templateapp = Flask(__name__)@app.route('/hello/<user>')def hello_name(user):   return render_template('hello.html', name = user)if __name__ == '__main__':   app.run(debug = True)
‘Web templating system’ refers to designing an HTML script in which the variable data can be inserted dynamically. A web template system comprises of a template engine, some kind of data source and a template processor.
Flask uses jinga2 template engine. A web template contains HTML syntax interspersed placeholders for variables and expressions (in these case Python expressions) which are replaced values when the template is rendered.
The following code is saved as hello.html in the templates folder.
<!doctype html><html>   <body>         <h1>Hello {{ name }}!</h1>         </body></html>
The Jinga2 template engine uses the following delimiters for escaping from HTML.
{% ... %} for Statements
{{ ... }} for Expressions to print to the template output
{# ... #} for Comments not included in the template output
# ... ## for Line Statements
The URL rule to the hello() function accepts the integer parameter. It is passed to the hello.html template. Inside it, the value of number received (marks) is compared (greater or less than 50) and accordingly HTML is conditionally rendered.
from flask import Flask, render_templateapp = Flask(__name__)@app.route('/result')def result():   dict = {'phy':50,'che':60,'maths':70}   return render_template('result.html', result = dict)if __name__ == '__main__':   app.run(debug = True)<!doctype html><html>   <body>         <table border = 1>         {% for key, value in result.iteritems() %}                     <tr>               <th> {{ key }} </th>               <td> {{ value }} </td>            </tr>                     {% endfor %}      </table>         </body></html>