Experimental Filerator Capabilities

This patch adds experimental "filerator" features

We are working on creating a final design for so called "filerator"
features -- iterators that support reasoning about files and
directories.  While we have not converged on a final design yet, this
commit provides initial support for some routines that approximate the
expected final filerator routines, for the purposes of experimentation
and feedback.

The commit breaks down as follows:

* It adds a new temporary/experimental module named 'Filerator' (the
  name is purposefully chosen to be playful to underscore the fact
  that it is experimental and will be renamed once we lock down the
  interface a bit harder.  This module supports four main iterators:

  1) iter listdir(path: string, dotfiles=false, dirs=true, files=true, 
                  listlinks=true): string

     listdir() lists the contents of a directory, similar to 'ls'

     * path: the directory whose contents should be listed
     * dotfiles: should files/directories that start with '.' be listed?
     * dirs: should dirs be listed?
     * files: should files be listed?
     * listlinks: should symbolic links be listed?

    By default this routine lists all files and directories in the
    current directory, including symbolic links, as long as they don't
    start with '.'

  2) iter walkdirs(path: string=".", topdown=true, depth=max(int),
                   dotfiles=false, followlinks=false, sort = false): string {

     walkdirs() recursively walks a directory structure, yielding
     directory names

     * path: the directory to start from
     * topdown: indicates whether to yield the directories using a
       preorder traversal
     * depth: indicates the maximal depth of recursion to use
     * dotfiles: indicates whether to enter directories with dotfile names
     * followlinks: indicates whether to follow symbolic links or not
     * sort: indicates whether to consider subdirectories in sorted order or not

     By default, walkdirs() will start in the current directory,
     process directories in preorder; recursively traverse
     subdirectories; and neither follow dotfile directories nor
     symbolic links.  It will not sort the directories by default.

  3) glob(pattern: string="*")

     glob() gives a glob-like capability, implemented with C's glob()

     * pattern: the pattern to match against

     By default, it will list all files/directories in the directory

  4) wordexp(pattern: string="*") 

     wordexp() gives a glob-like capability, implemented with C's wordexp()

     * pattern: the pattern to match against

     By default, it will list all files/directories in the directory

  Note that in the steady-state, only one of the final two are likely to
  be required.


* To support these iterators, functionality was added to the runtime
  in order to:

  (a) wrap C's dirent() (runtime/include/chpldirent.h)

  (b) provide an "is this a directory?" query implemented using
      stat()/lstat(), and (runtime/include/chplisdir.h,
      runtime/src/chplisdir.c)

  (c) wrap C's glob() and wordexp(). (runtime/include/chplglob.h)

  All new headers were added to stdchpl.h


* test/studies/filerator/ was updated and expanded to test the
  capabilities of the routines in a number of interesting
  configurations (but a by no means exhaustive set)


Just realized I never did anything about this one before committing, but 
only because I was multitasking too much.  Let me add it to my code freeze 
list and see if I can get to it (I did find that there are other places 
that this assumption is made, so it could also be captured as a broader 
pivotal task).

-Brad


On Tue, 9 Sep 2014, Thomas Van Doren wrote:

>> +
>> +    return chpl_rt_direntptr_getname(this);
>> +  }
>> +  // End inlining
>> +
>> +  var dir: DIRptr;
>> +  var ent: direntptr;
>> +  //  writeln("***Trying ", path);
>> +  dir = opendir(path:c_string);
>> +  if (!is_c_nil(dir)) {
>> +    ent = readdir(dir);
>> +    while (!is_c_nil(ent)) {
>> +      const filename = ent.d_name();
>> +      if (dotfiles || filename.substring(1) != '.') {
>> +        if (filename != "." && filename != "..") {
>> +          const fullpath = path + "/" + filename;
>
> Will you please factor the separator out into a const? Maybe something like `SEP` or `PATH_SEP` or ?
>
> ---
> Reply to this email directly or view it on GitHub:
> https://github.com/chapel-lang/chapel/pull/351/files#r17337478
>