NEWBIES, YOU'RE SCREWED (AKA, Forum Search doesn't work half the time) - Page 1
Forum section: electronics / general

author: Mark_O
author_rank: Frequent Contributor
timestamp: December 29, 2013, 02:53:33 am
content: The Topic formerly known as: NEWBIES, YOU'RE SCREWED (AKA, Forum Search doesn't work half the time)(revised Jan2/2014)Recently Davewrote:"It is quite likely that your question has already been answered in some form, or your product already discussed extensively, so please use the Search facility located in the menu system above as a first pass. Don't use the search box top right, it's not as good."This is good advice, and sometimes helpful. But while it's perfectly true that this forum holds a vast wealth of information and knowledge, locating it is not as simple as one might hope. Here's a simple example, triggered by a recent newbie request:Using the "good" Search function in the top menu, look for "DS1000Z firmware".You'll get a number of hits (10, in this case). What youwon'tget is a link to:https://www.eevblog.com/forum/testgear/sniffing-the-rigol%27s-internal-i2c-bus/msg345866/#msg345866which containsboth of the search terms explicitly, and is the only one providing a download link. For unknown reasons, the system won't show you that one.This is just one example I had at hand, but it happens fairly frequently. You just won't realize it unless you already know the post you're looking for, and notice a search isn't including it.IMO, this is a contributing factor to why we persistently see so many requests (by newbies and otherwise). Even if they do perform a Search, they're still missing out on vital info that non-newbies have seen and processed. Hell, I have trouble often myself, findingmy ownposts.There are many truly excellent things about this Forum, but unfortunately Search is not one of them.

author: xrunner
author_rank: Super Contributor
timestamp: December 29, 2013, 02:59:11 am
content: It might be the search method that's set up. I had a problem with that on my forum and fixed it by going to a full text index. The Admin here could look into it.Forum - Search methodSearch indexFulltext index

author: GeoffS
author_rank: Unknown
timestamp: December 29, 2013, 03:00:27 am
content: I agree that the se4arch function in SMF leaves something to be desired.Try using a google search specifying the site eevblog.com e.gDS1000Z firmware site: eevblog.comNot the perfect solution but certainly a work around.EDIT: I take that back, the above google search only returns 3 hits for this forum.

author: c4757p
author_rank: Super Contributor
timestamp: December 29, 2013, 03:08:58 am
content: Quote from: GeoffS on December 29, 2013, 03:00:27 amDelete the space after site:

author: robrenz
author_rank: Super Contributor
timestamp: December 29, 2013, 03:10:32 am
content: Search works fine and returns the one you wanted (18th item) you just need to set "matchanywords" instead of the default "matchallwords".Maybe the default can be changed to "match any words" by the moderators of Dave.

author: xrunner
author_rank: Super Contributor
timestamp: December 29, 2013, 03:11:25 am
content: Quote from: GeoffS on December 29, 2013, 03:00:27 amHello ... are you going to check into my advice above?

author: Mark_O
author_rank: Frequent Contributor
timestamp: December 29, 2013, 03:20:11 am
content: Quote from: robrenz on December 29, 2013, 03:10:32 amRob, thanks for the tip. But I wonder how you can say it "works fine", when all wordsare presentand it doesn't match on that post? With all due respect, that's not "working fine" in my book.While your tip may be helpful, it will also often result in an explosion of hits, from unrelated topics and posts. Which you then have to dig through manually. For some reason, I thought that was supposed to be the computer's job?

author: GeoffS
author_rank: Unknown
timestamp: December 29, 2013, 03:27:18 am
content: Quote from: c4757p on December 29, 2013, 03:08:58 amNew glasses time

author: EEVblog
author_rank: Unknown
timestamp: December 29, 2013, 03:28:09 am
content: Quote from: xrunner on December 29, 2013, 02:59:11 amDone.These are the other options, and the default values set:Relative search weight for number of matching messages within a topic: 30%Help Relative search weight for age of last matching message: 25%Help Relative search weight for topic length: 20%Help Relative search weight for a matching subject: 15%Help Relative search weight for a first message match: 10%Help Relative search weight for a sticky topic: 0% Total

author: liquibyte
author_rank: Frequent Contributor
timestamp: December 29, 2013, 03:28:29 am
content: Quote from: GeoffS on December 29, 2013, 03:00:27 amAlso:Code:inurl:www.eevblog.com +DS100Z +firmware

author: EEVblog
author_rank: Unknown
timestamp: December 29, 2013, 03:39:06 am
content: I've added a mod that adds a search button to boards and also topic threads.

author: xrunner
author_rank: Super Contributor
timestamp: December 29, 2013, 03:41:00 am
content: Quote from: EEVblog on December 29, 2013, 03:28:09 amCool. Even if it doesn't help this issue, I know it will improve searches. Your database is larger than mine, and my members were complaining some months ago that the searches were not returning many results. Once I changed that setting, it was like night and day as far as search results were concerned.

author: EEVblog
author_rank: Unknown
timestamp: December 29, 2013, 03:43:00 am
content: I'd like to install this, but it's not compatible with this version of the forumhttp://custom.simplemachines.org/mods/index.php?mod=887

author: van-c
author_rank: Regular Contributor
timestamp: December 29, 2013, 05:43:16 am
content: This problem is especially frustrating for long discussions. While in any discussionsubjectin the forum, using the Search button at the top right corner of the page OR the search button at the top of a reply section limits the search to that subject as expected, but it willneverlist more than the first 30 replies containing matches. For example, search for "firmware" in the subject "Sniffing the Rigol's Internal I2C bus;" you'll get only 30 matched replies, although there are many more than that in the discussion. Using the "Advanced" search will return more than 30 results, but only from topics and it seems only one or very few matched replies per Subject.I raised this issue in the General Chat area a couple of days ago and got one reply, suggesting to use Google search. Yeah, right. Google does not help at all when you're trying to find references within a long discussion subject.--Van

author: mrflibble
author_rank: Super Contributor
timestamp: December 29, 2013, 10:19:44 am
content: Quote from: van-c on December 29, 2013, 05:43:16 amWell, it works for me. I sometimes try to find a reference in the Flir E4 teardown thread (2625 posts as of now), and unfortunately the forum search function doesn't work all that well. However google search forsite:eevblog.com/forum Flir E4 <whatever_im_looking_for>works quite well.

author: KedasProbe
author_rank: Frequent Contributor
timestamp: December 29, 2013, 10:41:21 am
content: Since google seem to do a good job as second search engine, maybe the search textbox at the top right can be replaced with a google search texbox (limited to this forum)

author: EEVblog
author_rank: Unknown
timestamp: December 29, 2013, 10:56:48 am
content: Quote from: KedasProbe on December 29, 2013, 10:41:21 amYes, I'd love to know how to do this, anyone?

author: gmb42
author_rank: Frequent Contributor
timestamp: December 29, 2013, 12:12:25 pm
content: Maybe this:Custom Google Search

author: KedasProbe
author_rank: Frequent Contributor
timestamp: December 29, 2013, 12:58:52 pm
content: As example this website is using google seacrh engine at the top:http://www.spray.com/

author: Mark Hennessy
author_rank: Frequent Contributor
timestamp: December 29, 2013, 01:08:56 pm
content: It was added to this UK forum last year:http://www.vintage-radio.net/forum/index.php- the discussion about it is here:http://www.vintage-radio.net/forum/showthread.php?t=82628- and it does work very well indeed. Looking at the source code of the form, it looks to be very straightforward.

author: GeoffS
author_rank: Unknown
timestamp: December 29, 2013, 01:13:23 pm
content: Quote from: Mark Hennessy on December 29, 2013, 01:08:56 pmThis is for vBulletin though, not SMF as used here.There is a mod for SMF google searches but it's not for the latest version so is has to be patched in manually.

author: Andy Watson
author_rank: Super Contributor
timestamp: December 29, 2013, 01:17:41 pm
content: Quote from: KedasProbe on December 29, 2013, 10:41:21 amI'm pretty sure this would necessitate a call-home to Google telling them thatIam using this forum.

author: Mark Hennessy
author_rank: Frequent Contributor
timestamp: December 29, 2013, 01:25:52 pm
content: Quote from: GeoffS on December 29, 2013, 01:13:23 pmAre you sure? As far as I can understand - and from how it's explained on that thread I linked to - it's simply using Google Custom Search.Here is the source code, which I think has been manually added to the header source code.Code:<form id="cse-search-box" action="http://www.google.com/cse"><input type="hidden" name="cx" value="008872823631354711830:4sch4flwhxq" /><input type="hidden" name="ie" value="UTF-8" /><input type="text" name="q" id="q" /><br /><input type="submit" name="s" id="s" value="Search this website" /></form>As you can see, it's a bog-standard HTML form that sends the query to google.com - I assume it's the "cx" value that identifies that site to Google Custom Search?

author: xrunner
author_rank: Super Contributor
timestamp: December 29, 2013, 02:46:20 pm
content: Methinks the SMF search function is still not set up right. I searched for transistor, then resistor, then capacitor, and I always get backexactly30 results. That's not coincidence. Should be a whole bunch 'o results for those terms.Dave, could you check this -Admin - forum - search - settingsNumber of search results per page - ?Maximum results to show - ?(0: no limit)

author: Bored@Work
author_rank: Super Contributor
timestamp: December 29, 2013, 03:27:53 pm
content: Quote from: van-c on December 29, 2013, 05:43:16 amYes, right. I gave that suggestion. And if you don't manage to type in a simple google search then, well ... Have you considered a non-electronics hobby?

author: Mark Hennessy
author_rank: Frequent Contributor
timestamp: December 29, 2013, 03:35:24 pm
content: Quote from: Mark Hennessy on December 29, 2013, 01:25:52 pmOK - because I've been known to get things wrong, I thought I'd better check this out myself.I copied and pasted the code I quoted earlier into a page on my website, and found that it did indeed work. Anything I typed in returned a page of results from the UK vintage forum. That proved that Google isn't simply looking at the referring site before deciding which site is raising the query...Next, that funny business in the "cx" field:I changed a single character in that string, and that stopped the search from working. Good - as suspected, the long string in the "cx" field is indeed telling Google which site is being searched.So I decided to set up a Custom Search, using the URL provided by gmb42 earlier. That was dead-easy - literally just a case of typing in my site URL and clicking on "go". In return, I have been given a load of stuff that I can paste into my website to generate the Google search:Code:<script>(function() {var cx = '005351797449373264470:qjxoqys2r60';var gcse = document.createElement('script');gcse.type = 'text/javascript';gcse.async = true;gcse.src = (document.location.protocol == 'https:' ? 'https:' : 'http:')'//www.google.com/cse/cse.js?cx=' cx;var s = document.getElementsByTagName('script');s.parentNode.insertBefore(gcse, s);})();</script><gcse:search></gcse:search>But, this all looks a bit complex, so I haven't tried that yet. Instead, I took the HTML that I posted earlier, and simply replaced the "cx" value with the one given to me in the above code. And it works!All of that took about 5 minutes. Adding something like this to the forum ought to be dead simple - it's just a case of understanding how SMF organise their files/templates/whatever, so that you know which file needs to receive the HMTL to put the search box onto every page...Good luck!Mark

author: diynoise
author_rank: Contributor
timestamp: December 29, 2013, 03:52:25 pm
content: I think that to install that simple HTML code on a smf forum you need a mod, though. At least for it to not cause problems when you update the forum, otherwise you could just edit the theme.But don't quote me on that, that's what I remember from when I was an admin at a smaller smf forum.

author: Mark Hennessy
author_rank: Frequent Contributor
timestamp: December 29, 2013, 04:02:14 pm
content: Can't speak for SMF, but I've done this sort of thing on MyBB and it was trivially easy. Obviously you're at the mercy of the people who code SMF - it's their ecosystem - but I can't believe it's that difficult to anyone who understands PHP. If I get a bit of spare time later I'll download it and play with it...

author: xrunner
author_rank: Super Contributor
timestamp: December 30, 2013, 01:21:33 pm
content: Has an Admin looked into what I said below? The search only returns 30 results and that's not the way it's 'posed to work.Quote from: xrunner on December 29, 2013, 02:46:20 pm

author: van-c
author_rank: Regular Contributor
timestamp: December 30, 2013, 04:40:30 pm
content: Quote from: xrunner on December 30, 2013, 01:21:33 pmI've mentioned this 30 results limit too. When you're searching for a specific case of a very common topic, having all occurrences returned is crucial, even when the number returned is very large. I do like the way all cases of the search string are highlighted, though. The way results are returned by the forum search tool makes finding a particular item of interest embedded in a long list of results a lot easier, IMO, than trudging through many Google results.If the tool would just return all occurrences without limiting the number, that would help a great deal.

author: Unknown
author_rank: Unknown
timestamp: January 01, 2014, 03:31:03 pm
content: This thread is in the wrong place.

author: gnif
author_rank: Unknown
timestamp: January 02, 2014, 02:15:56 pm
content: Quote from: xrunner on December 29, 2013, 02:59:11 amThis should be profiled before just enabling it, this adds 'FULLTEXT' indexes into the database which can hurt performance badly with forums as large as this. Also some of the database tables have been converted to InnoDB to correct performance issues we were having some time ago, InnoDB does not support FULLTEXT indexes (yet). A better option would be to look at a proper dedicated indexer such as Sphynx, I would be surprised if someone has not made a Sphynx plugin for SMF. Failing that, Google is your friend.

author: xrunner
author_rank: Super Contributor
timestamp: January 02, 2014, 02:41:46 pm
content: Gnif, the real issue is that the search returns only 30 results, when it is obvious, such as searching for "transistor" or "resistor" that there are far more than 30 occurrences of that text in the database. We were simply troubleshooting that issue to see if anything had an effect on that. Can you offer any insight to the crux of the issue, as I've just now explained? I have been doing some investigating on the SMF forum but have not found the definitive answer yet.Oh and yes, this thread is in need of splitting.

author: SeanB
author_rank: Super Contributor
timestamp: January 02, 2014, 02:42:33 pm
content: How about changing the search to use Google in the top bar, with as interim window warning that you are going to the external site. line could still use the default method.

author: gnif
author_rank: Unknown
timestamp: January 02, 2014, 02:54:10 pm
content: Quote from: xrunner on January 02, 2014, 02:41:46 pmThe reason the number of search results are limited is to prevent overloading of the server, text searches are very processor and I/O intensive... if it was allowed to return everything in one go, a single user could effectively DOS the forums simply by performing a few searches. I am looking at Sphynx, seems that SMF has support for it, I need to set-up a test server and have a play and see how it performs before even thinking of deploying it. I will discuss this with Dave over the coming days, I may even take a read-only copy of these forums as a good test sample set and give a few people access to trial the search functionality.

author: xrunner
author_rank: Super Contributor
timestamp: January 02, 2014, 02:57:34 pm
content: Quote from: gnif on January 02, 2014, 02:54:10 pmOh OK, so you are doing iton purpose. Well nobody said that until now. From what Dave was seeming to say when we were playing with it, he didn't think the 30 result thing was acting properly either. So that's where the confusion lies. I'm not sure that it can't really handle returning more than 30, such as perhaps 100 or so, but it's your call.

author: gnif
author_rank: Unknown
timestamp: January 02, 2014, 03:01:33 pm
content: Quote from: xrunner on January 02, 2014, 02:57:34 pmI did not set the limit, it is whatever Dave has configured it at, I am not sure why 30 was chosen, could just be the default. The better solution though would be to remove the limit completely and move to a proper indexer as this solution is both scalable and future proof (read: wont cause a slowdown as the forum gets bigger and bigger)

author: marmad
author_rank: Super Contributor
timestamp: January 02, 2014, 03:04:34 pm
content: Quote from: gnif on January 02, 2014, 02:54:10 pmBut this is assuming it's working in some logical way - but as already reported, it isn't. When you use advanced search with no restrictions, you get back a single page of 30 results. When you use advanced search with "Search in topic subjects only" (ostensibly limiting the search), you get back multiple pages of 30 results.

author: gnif
author_rank: Unknown
timestamp: January 02, 2014, 03:06:34 pm
content: Quote from: marmad on January 02, 2014, 03:04:34 pmThus the reason for moving to a more advanced search engine... the SMF simple search is just too primitive for the size of this forum now.

author: xrunner
author_rank: Super Contributor
timestamp: January 02, 2014, 03:58:05 pm
content: Quote from: gnif on January 02, 2014, 03:06:34 pmOK gnif that's cool, but I hope you can do that. If you want this to be a valuable resource of technical info (and it is) then people need to perform research with a good return of results. The answer they want might be search result 31 but never to be seen ...Good Luck.