ls-files: buffer full item in strbuf before printing

Buffering so that we can manipulate the strings (e.g. coloring)
further before finally printing them.
ls-files: add --color to highlight file names

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
ls-files: add --column
ls-files: support --max-depth
ls-files: split main ls-files logic into ls_files() function

This is a preparation step for the introduction of git-ls. "git ls"
has a different set of command line options, but it will eventually
call ls_files().
Add git-ls, a user friendly version of ls-files and more

This is more user friendly version of ls-files:

* it's automatically colored and columnized
* it refreshes the index like all porcelain commands
* it defaults to non-recursive behavior like ls
* :(glob) is on by default so '*.c' means a.c but not a/b.c, use
  '**/*.c' for that.
* auto pager
ls: -u does not imply showing stages

Showing full index entry information is something for ls-files
only. The users of "git ls" may just want to know what entries are not
unmerged.
ls: add -R/--recursive short for --max-depth=-1
ls: add -1 short for --no-column in the spirit of GNU ls
ls: add -t back

Tag "H" (cached) is not shown though because it's usually the majority
and becomes noise. Not showing it makes the other tags stand out. -t
is on by default if more than one file category is selected.
ls: sort output and remove duplicates

When you mix different file types, with ls-files you may get separate
listing. For example, "ls-files -cm" will show file "abc" twice: one
as part of cached list, one of modified list. With "ls" (and this
patch) they will be in a single sorted list (easier for the eye).

Duplicate entries are also removed. Note that display content is
compared, so if you have "-t" on, or you color file types differently,
you will get duplicate textual entries. This is good imo.
ls: do not show duplicate cached entries

With the current show_files() "ls -tcm" will show

  foo.c
M foo.c

The first item is redundant. If "foo.c" is modified, we know it's in
the cache. Introduce show_files_compact to do that because ls-files is
plumbing and scripts may already depend on current display behavior.

Another difference in show_files_compact() is it does not show
skip-worktree (aka outside sparse checkout) entries anymore, which
makes sense in porcelain context.
ls: show directories as well as files

The index does not store directories explicitly (except submodules) so
we have to figure them out from file list. The function
show_directories() deliberately generates duplicate directories and
expects the previous patch to remove duplicates.
ls: -M aka diff-cached
column: support "denser" mode

Sometimes a few long entries in the listing would stretch out columns,
wasting space. If these entries are cut out, the columns could become
smaller and more columns can be displayed.

This mode checks the shortest and longest entries of a range of
items. If the difference is over a limit, the range is cut out for
displaying. This makes sure a group or long file names won't break
make any columns too wide.

Try "git ls -R t" with --column=denser and --column=dense to see the
difference.