api-directory-listing.txt: update to match code

7c4c97c0ac turned the flags in struct dir_struct into a single bitfield
variable, but forgot to update this document.

Signed-off-by: Adam Spiers <git@adamspiers.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Improve documentation and comments regarding directory traversal API

traversal API has a few potentially confusing properties.  These
comments clarify a few key aspects and will hopefully make it easier
to understand for other newcomers in the future.

Signed-off-by: Adam Spiers <git@adamspiers.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
dir.c: rename cryptic 'which' variable to more consistent name

'el' is only *slightly* less cryptic, but is already used as the
variable name for a struct exclude_list pointer in numerous other
places, so this reduces the number of cryptic variable names in use by
one :-)

Signed-off-by: Adam Spiers <git@adamspiers.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
dir.c: rename path_excluded() to is_path_excluded()

Start adopting clearer names for exclude functions.  This 'is_*'
naming pattern for functions returning booleans was agreed here:

http://thread.gmane.org/gmane.comp.version-control.git/204661/focus=204924

Signed-off-by: Adam Spiers <git@adamspiers.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
dir.c: rename excluded_from_list() to is_excluded_from_list()

Continue adopting clearer names for exclude functions.  This 'is_*'
naming pattern for functions returning booleans was discussed here:

http://thread.gmane.org/gmane.comp.version-control.git/204661/focus=204924

Also adjust their callers as necessary.

Signed-off-by: Adam Spiers <git@adamspiers.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
dir.c: rename excluded() to is_excluded()

Continue adopting clearer names for exclude functions.  This is_*
naming pattern for functions returning booleans was discussed here:

http://thread.gmane.org/gmane.comp.version-control.git/204661/focus=204924

Signed-off-by: Adam Spiers <git@adamspiers.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
dir.c: refactor is_excluded_from_list()

The excluded function uses a new helper function called
last_exclude_matching_from_list() to perform the inner loop over all of
the exclude patterns.  The helper just tells us whether the path is
included, excluded, or undecided.

However, it may be useful to know _which_ pattern was triggered.  So
let's pass out the entire exclude match, which contains the status
information we were already passing out.

Further patches can make use of this.

This is a modified forward port of a patch from 2009 by Jeff King:
http://article.gmane.org/gmane.comp.version-control.git/108815

Signed-off-by: Adam Spiers <git@adamspiers.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
dir.c: refactor is_excluded()

In a similar way to the previous commit, this extracts a new helper
function last_exclude_matching() which returns the last exclude_list
element which matched, or NULL if no match was found.  is_excluded()
becomes a wrapper around this, and just returns 0 or 1 depending on
whether any matching exclude_list element was found.

This allows callers to find out _why_ a given path was excluded,
rather than just whether it was or not, paving the way for a new git
sub-command which allows users to test their exclude lists from the
command line.

Signed-off-by: Adam Spiers <git@adamspiers.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
dir.c: refactor is_path_excluded()

In a similar way to the previous commit, this extracts a new helper
function last_exclude_matching_path() which return the last
exclude_list element which matched, or NULL if no match was found.
is_path_excluded() becomes a wrapper around this, and just returns 0
or 1 depending on whether any matching exclude_list element was found.

This allows callers to find out _why_ a given path was excluded,
rather than just whether it was or not, paving the way for a new git
sub-command which allows users to test their exclude lists from the
command line.

Signed-off-by: Adam Spiers <git@adamspiers.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
dir.c: rename free_excludes() to clear_exclude_list()

It is clearer to use a 'clear_' prefix for functions which empty
and deallocate the contents of a data structure without freeing
the structure itself, and a 'free_' prefix for functions which
also free the structure itself.

http://article.gmane.org/gmane.comp.version-control.git/206128

Signed-off-by: Adam Spiers <git@adamspiers.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
dir.c: use a single struct exclude_list per source of excludes

Previously each exclude_list could potentially contain patterns
from multiple sources.  For example dir->exclude_list[EXC_FILE]
would typically contain patterns from .git/info/exclude and
core.excludesfile, and dir->exclude_list[EXC_DIRS] could contain
patterns from multiple per-directory .gitignore files during
directory traversal (i.e. when dir->exclude_stack was more than
one item deep).

We split these composite exclude_lists up into three groups of
exclude_lists (EXC_CMDL / EXC_DIRS / EXC_FILE as before), so that each
exclude_list now contains patterns from a single source.  This will
allow us to cleanly track the origin of each pattern simply by adding
a src field to struct exclude_list, rather than to struct exclude,
which would make memory management of the source string tricky in the
EXC_DIRS case where its contents are dynamically generated.

Similarly, by moving the filebuf member from struct exclude_stack to
struct exclude_list, it allows us to track and subsequently free
memory buffers allocated during the parsing of all exclude files,
rather than only tracking buffers allocated for files in the EXC_DIRS
group.
[SQUASH] avoid assuming knowledge of the internal implementation of exclude_list_groups

On Sun, Jan 06, 2013 at 12:25:48PM -0800, Junio C Hamano wrote:
> Adam Spiers <git@adamspiers.org> writes:
>
> > On Fri, Jan 04, 2013 at 01:03:59PM -0800, Junio C Hamano wrote:
> >> Adam Spiers <git@adamspiers.org> writes:
> >>
> >> > diff --git a/builtin/clean.c b/builtin/clean.c
> >> > index 0c7b3d0..bd18b88 100644
> >> > --- a/builtin/clean.c
> >> > +++ b/builtin/clean.c
> >> > @@ -97,9 +97,10 @@ int cmd_clean(int argc, const char **argv, const char *prefix)
> >> >  	if (!ignored)
> >> >  		setup_standard_excludes(&dir);
> >> >
> >> > +	add_exclude_list(&dir, EXC_CMDL);
> >> >  	for (i = 0; i < exclude_list.nr; i++)
> >> >  		add_exclude(exclude_list.items[i].string, "", 0,
> >> > -			    &dir.exclude_list[EXC_CMDL]);
> >> > +			    &dir.exclude_list_groups[EXC_CMDL].ary[0]);
> >>
> >> This looks somewhat ugly for two reasons.
> >>
> >>  * The abstraction add_exclude() offers to its callers is just to
> >>    let them add one pattern to the list of patterns for the kind
> >>    (here, EXC_CMDL); why should they care about .ary[0] part?
> >
> > Because the caller has to decide which exclude list the new exclude
> > should be added to; that is how it has been for a long time, and I am
> > not proposing we change that.
>
> Unless I was mistaken, I never objected to the EXC_CMDL, etc
> appearing in the text of the calling site of add_exclude().
>
> The objection was about the .ary[0] bit.  From the point of view of
> a caller of the API, it:
>
>     - calls add_exclude_list() to declare "I now start adding new
>       patterns that come from a new source of patterns"; then
>
>     - calls add_exclude() repeatedly to add the patterns that come
>       from that source.
>
> no?

Correct.

> Why does the latter has to keep repeating "Here is the new
> pattern for the EXC_CMDL group; it comes from the latest source I
> earlier declared, by the way", instead of just "Here is the new
> pattern for the EXC_CMDL group"?

Mainly because there is no guarantee that such a group exists.

unpack_trees() has:

	if (add_excludes_from_file_to_list(git_path("info/sparse-checkout"), "", 0, &el, 0) < 0)

so if you change the signature of add_exclude() to require an
exclude_list_group, then there is no way to implement
add_excludes_from_file_to_list().

Even if you could, you still haven't reduced the number of parameters
add_exclude() requires, so I'm dubious of the benefits of this
"simplification".

> >>    Are
> >>    there cases any sane caller (not the implementation of the
> >>    exclude_list_group machinery e.g. add_excludes_from_... function)
> >>    may want to call it with .ary[1]?
> >
> > Effectively yes, although it is not written like ".ary[1]".  For
> > example prep_exclude() calls add_excludes_from_file_to_list() for each
> > new .gitignore file
>
> That is part of the "implementation of the machinery".  If the API
> for the outside callers are to call add_exclude_list() to declare
> that patterns added by subsequent calls to add_exclude() are from
> one new source of the patterns (e.g. .gitignore file in a new
> directory level), and then call add_exclude() to add each pattern,
> then the callers to add_exclude() shouldn't have to care about the
> implementation detail that individual sources in exclude_list_group
> is implemented as an array in that sructure, and the latest ones
> should go to its ->array[0].

That's a valid point.  However, the ary[0] part which assumes external
knowledge of the internal implementation can trivially be avoided by
squashing this patch onto the commit we are discussing:
dir.c: keep track of where patterns came from

For exclude patterns read in from files, the filename is stored in the
exclude list, and the originating line number is stored in the
individual exclude (counting starting at 1).

For exclude patterns provided on the command line, a string describing
the source of the patterns is stored in the exclude list, and the
sequence number assigned to each exclude pattern is negative, with
counting starting at -1.  So for example the 2nd pattern provided via
--exclude would be numbered -2.  This allows any future consumers of
that data to easily distinguish between exclude patterns from files
vs. from the CLI.
dir.c: provide clear_directory() for reclaiming dir_struct memory

By the end of a directory traversal, a dir_struct instance will
typically contains pointers to various data structures on the heap.
clear_directory() provides a convenient way to reclaim that memory.
dir.c: improve docs for match_pathspec() and match_pathspec_depth()

Fix a grammatical issue in the description of these functions, and
make it more obvious how and why seen[] can be reused across multiple
invocations.
add.c: remove unused argument from validate_pathspec()

The 'argc' argument passed to validate_pathspec() was never used.
add.c: move pathspec matchers into new pathspec.c for reuse

Extract the following functions from builtin/add.c to pathspec.c, in
preparation for reuse by a new git check-ignore command:

  - fill_pathspec_matches()
  - find_used_pathspec()

The functions being extracted are not changed in any way, except
removal of the 'static' qualifier.

Also add comments documenting these newly public functions,
including clarifications that they operate on the index.
pathspec.c: rename newly public functions for clarity

Perform the following function renames to make it explicit that these
pathspec handling functions are for matching against the index, rather
than against a tree or the working directory.

- fill_pathspec_matches() -> add_pathspec_matches_against_index()
- find_used_pathspec() -> find_pathspecs_matching_against_index()
add.c: extract check_path_for_gitlink() from treat_gitlinks() for reuse

Extract the body of the for loop in treat_gitlinks() into a separate
check_path_for_gitlink() function so that it can be reused elsewhere.
This paves the way for a new check-ignore sub-command.

Also document treat_gitlinks().
add.c: extract new die_if_path_beyond_symlink() for reuse

This will be reused by a new git check-ignore command.

Also document validate_pathspec().