Repository: bokeh/bokeh

Issue:
[DOCS] Page wise display of documentation search 
**Is your feature request related to a problem?**
Yes. I searched for a relatively simple query in the documentation search bar of https://docs.bokeh.org, and it took very long to load the results. In my second try, the results weren't even loading, I'm afraid. These are displayed in an unordered list which fills the entire page up. It might get frustrating to read through everything to find the answer to the input query.   

**Describe the solution you'd like**
I would suggest displaying the fetched results in a page wise format, the way most search engines do it. Relevance weighted sorted answer, shown page wise. Fill up only the current page of about 20 to 30 odd query results, and depending on whether the user wants to see the other pages, load them.

**Describe alternatives you've considered**
If not a page wise result, a folder wise result would also benefit, which leaves the option to the user to navigate where he/she wants to. A custom google search may also help.

**Additional context**
![issue](https://user-images.githubusercontent.com/43414361/82223644-88c83400-9940-11ea-9534-f96dc4ae3a6d.PNG)




Discussion:
Hi @nimishbongale thanks for the issue. I want to preface by saying that I hope my reply does not come off as overly pessimistic. I would love to see search UX improve, but I think the options in the short term may be limited. In any case I wanted to provide some context and history around how we got where we are, as well as what I see as possible future options.

---

### History

First off, I should not that there was no built-in search on the docs at all for maybe ~5 years, so the current situation is clearly an improvement over that :) Additionally I have not run in to the issue of a search not re-running a second time. I do agree that the results "over-report" but at least from what I have seen, the relevant search result is usually right near the top, and the results do progressively load, which mitigates the overly long list. Overall I would say things as they are have been "tolerable" but I appreciate that you had a different experience than me (and it may vary for folks depending heavily on browser, hardware, and connection)

#### Past 

The docs started off with a *Google Search Site* integration. This was affordable, but also not without some issues. I don't think we ever got it handling our multiple deployed versions well. However, it did not really matter. Google did it's usual thing and in 2016 announced that they were shutting GSS down. Everyone could move to a GCSE but the free tier is ad-supported and we could not afford a "real" one. We did not want to tolerate ads on the project docs to we simply removed search. 

Last year someone suggested [Algolia](https://www.algolia.com/) which is a very powerful search service product. Unfortunately it is exorbitantly expensive and far outside our budget. They do offer a free [Docsearch](https://docsearch.algolia.com/) service for OSS projects, but unfortunately free Docsearch results are really not good. The limit to only 5 results *total*, and those results are often garbage. (E.g. I searched for "DOM" on the React site and got noteve a single relevant return) I did discover that the things could be tweaked or worked around in JS by using the "real" Algolia APIs but I guess this was not intended usage. Once I tried to customize and improve a docsearch engine for Bokeh, the helpful support folks I'd been in contact with quickly became unhelpful (even unfriendly). So that idea was abandoned. 

#### Present

Currently we use the basic client-side search built in to [Sphinx](https://www.sphinx-doc.org/en/master/) (the docs tool). This generates a big JSON index during the docs build and we upload it to CDN with the rest of the docs when releases happen. It's fairly large, which is a downside, but also should be cached in browsers locally for a long time. One of the main benefits of this approach is that sharding by version is automatic and simple. If you search on the 2.0.2 page, you only get results for the 2.0.2 version. This is important, and, it seems, hard to get right with index services. It's also free, which is also important, tho a little less so than it used to be. However, it has some of the drawbacks you point out. 

#### Future

So what are some ideas for the future?

*  Improve the client search

   The current client-side search is just some JS that loads and looks through the index file. We could certainly modify it however we like. I'm not sure how paging would work in this case without making each successive page take longer to load (since it would just have to start over, I think) but maybe there is some clever way. Or maybe we just start with showing "top 10 results" and then a link goes to "full results", which is just the same as the current page. 

* Use GCSE

   We were able to obtain a for-nonprofits GCSE without ads through NumFOCUS recently. We stopped pursuing it because the client-side search was working "OK" and there were lots of other priorities. But you can check it out here, even:

    https://cse.google.com/cse?cx=000475829798580648034:1lwbjohiujq

    As you can see, it mixes results from different versions. So first and foremost, someone would need to figure out how to fix that. There's also a question of how to automatically trigger re-indexing when releases happen. 

* Run our own indexer / engine

  The Algolia Docsearch thing can apparently be run by hand, so we could spin up an EC2 instance to run our own search service. I will say, the full Algolia capabilities are *really slick*. I would love to be able to offer the kind of fancy faceted and filterable results they offer. But this is obviously the most work (and cost)

   Besides Algolia, we could also run our own elastic search, use use another service such as Sphinx (which is confusingly, completely unrelated to the Sphinx docs tool). 

   I think we could find  budget for something like this these days, but all of these options would really require a champion to step up to do the work. 

### Summary

I'd love to see this improve. I do think the best chance of that is with someone motivated who wants to "own" the search experience for the Bokeh docs. I think it's also worth drawing out a few requirements:

* able to handle version sharding
* mostly or fully automatable, so updates can happen from release machinery
* fits within our fairly modest budget

cc @bokeh/dev
Hello @bryevdv, this is some important information indeed!

Having a search feature is better than nothing, will certainly agree. 

I was stunned to see the amount of options that were considered for carrying out this search, and the limitations of each one of them. Given all of these points, I can certainly say your best bet would be the currently implemented one. 

But I do believe, there's some room for improvement; for some sort of an upgrade, if you may call it so. You see, I'm a front end person, and it really bothers me to see a page keep working on something and deliver me a rather lengthy unordered list at the end.  

The option to show 10 queries, and then to show the rest is also a reasonable solution (since the answer to the users query is most likely going to be in the first few queries itself). Maybe we could go a step further, and insert these "button breaks" for every 10-20 queries. I haven't delved deep into how all of this could be achieved, but I'm sure there is a way (like you mentioned).

The Algolia option is indeed great looking, but yes, It does involve a lot of work. In this case we'd really have to think if it's indeed necessary to do all of it. I'll admit, If I had to search for some function/feature in a library, I would just type "Bokeh get hyperplane slope and intercept" (or some equivalent feature) on Google and there would be many links including the documentation links as well. 

I never knew GCSE faced issues like this. I've seen on many websites, so I thought it is an option to consider, 

For now, I guess fixing this issue with method #1 is feasible, and improves the UX greatly. I'd be willing to figure out how this could be done. Will raise a PR if I'm able to do so! 

Thank you so much!
@nimishbongale the first option is definitely the quickest route to an incremental improvement for sure. AFAICT the relevant script is here:

https://docs.bokeh.org/en/latest/_static/searchtools.js

and also AFAICT the sphinx library automatically dumps it in the built static directory since I can't find it anywhere in our tree. However, I think if we drop a custom version in `sphinx/source/bokeh/static` then the custom version will take precedence. And even if that does not do it, I am sure there is *some* way. However for local testing you can just edit what is in `sphinx/build/html/_static` directly, if you want to try out any experiments. 

> I never knew GCSE faced issues like this.

I am fairly sure GCSE can handle multiple version fine with the appropriate configuration, but the configuration is confusing and I never figured out how to make it work better before moving on to other things. 
 @bryevdv  Alright, will look into the code!

True, rather get something working than indulge in more confusion ridden things!

Thank you!
Hello @bryevdv 

I've bee trying to get sphinx up and running for testing purposes, however, I find myself in a bit of a struggle. So much so that once I'm able to resolve these kind of issues, I'm sure I'll put it up publicly on the gsod thread for everyone to understand. 

If I've got everything right, I should be running `make html` to get the production files in the dist folder, Once I'm done with that, I should run the docserver to start the flask app, and navigate to localhost:5009.

All this is fair and good, but I'm unable to get the production HTML files. The bat file never runs completely, and I get errors such as 

> bokeh.models.markers has no attribute CircleDot

(or similar errors)

I can provide the logscript if that's required. I just want to make sure what I'm doing is right.

I looked through the JS which fetches the results of the query, and I'm sure once I'm able to get the app running, I can make considerable progress. 

Thanks for your time! 
> bokeh.models.markers has no attribute CircleDot

That model is very new, only added maybe a week ago, and not yet in any release. The only way I can imagine to get this message is that your docs build is importing an installed, previously released version of Bokeh (i.e. without recent additions), rather than a development version. Did you run `setup.py install` or `setup.py develop` in this python env?
@bryevdv 

I'm fairly confident I have run it, However I think I now realise what and why the error is arising.

In my system, the python alias was set to Python2.7 by default (required for react-native dev). I had installed bokeh via pip (for trying it out). Then, to run sphinx I went ahead and installed that as well. The `sphinx-build` command (by default) runs on Python2.7 (in my case, and there's a fix for it if someone wants to run it on Python3, which did eventually work). It searched for sphinx-build in `Python27/lib` and upon finding it there, went ahead with the build. I had a Python version conflict, and the "pip installed" version had a conflict with the latest updated version of bokeh as well.

I'm now certain what the error is, thanks to your response. I'll make a fresh venv, scrap the old repo I was working with, and pull the recent changes. 

Thanks again!   

Resolving patch:
diff --git a/sphinx/docserver.py b/sphinx/docserver.py
--- a/sphinx/docserver.py
+++ b/sphinx/docserver.py
@@ -1,3 +1,4 @@
+import asyncio
 import os
 import sys
 import threading
@@ -10,6 +11,11 @@
 from tornado.ioloop import IOLoop
 from tornado.wsgi import WSGIContainer
 
+# Needed for Windows + Python 3.8 config
+if sys.version_info.major==3 and sys.version_info.minor >= 8 and sys.platform.startswith('win'):
+    asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
+
+
 _basedir = os.path.join("..", os.path.dirname(__file__))
 
 app = flask.Flask(__name__, static_folder="/unused")


Test patch: