ajaxorg/cloud9 — pull request #2369 by gjtorikian (2012-10-30 05:32:13+00:00)
Title: Drop find and grep in favor of ag / nak

This PR represents a culmination of various 20% time I took over the last few weeks. It is designed to drop support for `find`/`grep` for filelist and search. Pull up a chair and a cup of coffee while I describe a change for a feature no one asked for. :)

## Why?

`find`/`grep` should be dropped for the following reasons:

1. The code is hard to maintain. (I should know, I wrote or worked on a lot of it.)
The code is a mess because we have to account for several annoyances, such as BSD vs. GNU flavors, bash-strings-as-scripts passing (which requires all sorts of escaping functions), and, in the case of `find`/`grep` missing, we actually compile PCRE support to provide our own binaries.
2. There's no Windows support. Conceivably, a user could SSH into a Windows box and run Cloud9. Dependence on `find`/`grep` limits this.
3. There's no easy way to maintain a list of accepted file types and ignored directories (see below for more). For example, we aren't overlooking _*-backup~_ from sm or _.c9revisions_.
4. There's no support for *users* to define their own search settings or exclusions.
5. There's no support (or _simple_ support) for files without extensions.
Want to search a Makefile, or a script starting with `#!/usr/bin/env`? Our current method of assembling in Javascript, writing a grep search, and tossing it over to the system via `bash -c` isn't supporting this. Good luck figuring out where to start.
6. It's slow--slow enough to not overlook all the other deficiencies listed above.

At first, I was pretty keen on just supporting [ack](betterthangrep.com) to overcome all of this. It's written in Perl, it supports ignore files, and it should be good enough, right?

Well, not quite.

## Advantages and Problems with Ack

ack 1.96, which is the current stable version, maintains a list of "safe" code files (with and without extensions). In fact, the current searchinfiles code was designed to emulate ack's behavior, and contains a list of known extensions to use.

ack 2.0 will move beyond using extensions to define files. This makes complete sense: every day another new technology drops, and no one wants to keep maintaining a list to add items like _.jade_, _.scss_, _.styl_ and so on. So, ack 2.0 uses Perl's [`-B`](http://stackoverflow.com/questions/899206/how-does-perl-know-a-file-is-binary) switch to determine if a file is binary. If it's not, it's searched. Perfect.

Unfortunately, ack 2.0 is alpha, and has been "under development" for around two years. As such, it doesn't really hold up to its promises of just searching non-binary. On top of all that, relying on ack for the filelist module wouldn't work, because it really doesn't even *list* binary files--it just outright ignores them.

Rather than hack up a solution in Perl, I went and chose something else.

## About Ag

[ag](https://github.com/ggreer/the_silver_searcher) is written in C. It's fast. It supports all of the above mentioned requirements (can list binaries, but only searches text; allows users to provide their own search settings; supports Windows, _e.t.c._), and can be used to replace both `find` and `grep`.

Perfect! I figured I would just use ag, and, for replacing files, continue our technique of [just using Perl to replace file contents](https://github.com/ajaxorg/cloud9/blob/master/plugins-server/cloud9.ide.search/search.js#L144-149).

Then, I had a panic attack. What if, _conceivably_, a user's machine didn't have Perl? They'd be SOL. So I needed another solution for replacing contents in files that didn't use Perl.

## Enter Nak

After reading Felix's [Faster than C](https://github.com/felixge/faster-than-c) notes, I became inspired to just write a *fast* ack clone, in Node.js. After all, if a user is SSHing into a box, it is guaranteed that they have Node.js installed.

Previously, TJ had [written an ack clone in Node](https://github.com/visionmedia/search), but his code was not very performant. At least, it was slower than ack.

I benchmarked and rewrote and learned a lot. I created [nak](https://github.com/c9/nak) which, while not supporting _everything_ ack does, does nearly everything ag does, at least, and 100% supports everything we need for Cloud9.

## Benchmarks

You like numbers? Me too. They're fun.

Here's the average time for grabbing the filelist in cloud9infra five times (about 33,761 files). The commands do the exact same thing (get hidden files, _e.t.c._) _and_ exclude the same nonsense directories (_.git_, _.c9revisions_, `sm` backups, _e.t.c._). :

`ag` | `nak` | `ack` | `find`
---------|----------|----------|---------
0m0.184s | 0m0.940s | 0m1.103s | 0m1.093s

Here are benchmarks for finding the phrase "va" in cloud9infra:

`ag` | `nak` | `ack` | `grep`
---------|----------|-----------|---------
0m2.599s | 0m20.021s| 0m29.876s | 0m20.891s

What ends up happening is that, with ag, the filelist comes up _before the IDE even finishes initializing_. Similarly, search results return _before the panel finishes animating up_. It's some impressive shit.

## Ignoring Files

There are now three ways to ignore files:

1. With a standard Cloud9 IDE _.agignore_ file. This will be loaded for every filelist and searchinfiles command. It follows gitignore-style wildcards. It excludes uninteresting directories, minified JS files, and Cloud9 extraneous content. This makes mainting our own rule-list much, much easier.

2. Users can define their own _.agignore_ on a per-directory basis in their workspaces.

3. While searching, users can exclude filetypes with `-`; for example, `-*_test.js` excludes all Javascript files ending with "_test.js".

## Testing

Mike did a huge favor by rewriting the tests to be compatible with Mocha. I extended his test suite, and also duplicated the commands, one for ag and one for nak; each test also validates that the results of ag and nak are the exact same.

## Final Thoughts

My idea is that we provide various binaries for ag, and, in the case that we don't support a platform, we fall back to nak.

By my count, we're missing the following binaries:

* freebsd
* sunos
* win32

In any event, though, all these binaries are built on an 64-bit machines, and if I learned anything from working on Cloud9 Local, it's that people are still using 32-bit machines (I tried to bundle x64 Node binaries for users without Node on their machine, and it wouldn't work for those running x32).

We won't really need to continue dropping new ag binaries, since it's currently fast enough, and, through the mocha tests, we've already covered every feature we need at the moment.

A more real problem, though, is that I'm not at all sure if one Linux binary (built on Ubuntu) will support other Linux distros (Fedora, CentOS, Gentoo...). If that's the case, then we should definitely fall back to nak. I couldn't figure out a way--using `process` or the `os` module--to see if we could specifically identify the type of the Linux distro.

Finally, it'd be great to also pull in these two PRs, which rename the packed files to end in .min.js:

* https://github.com/ajaxorg/cloud9infra/pull/2898
* https://github.com/ajaxorg/cloud9/pull/2312

I had a lot of fun writing performany Node code. Hooray!

@ajaxorg/liskov

@lennartcl @mostafaeweda @nightwing (as seen above, I am pretty sure non-team members cannot @ a team).