Repository: sharkdp/fd
Pull request #575 (2020-04-22 07:16:38)

Add global fdignore support
Rust newbie here :-)

This implements global fdignore support in `~/.config/fd/ignore` as mentioned [here](https://github.com/sharkdp/fd/issues/485#issuecomment-532869261).

I have some questions:
- I'm using `std::env::home_dir()` (which is deprecated) instead of using the `dirs` crate, because this is what the `ignore` crate does. Which would you prefer?
- I'm not too sure how to handle the tests. If I have, say, `*` in my `~/.config/fd/ignore`, the tests will break. The simplest way I can think of to deal with this is to add a `!*` using `--ignore-file`.
- Are there parts of the doc I should change?

sharkdp: > Rust newbie here :-)

Welcome!

> This implements global fdignore support in `~/.config/fd/ignore` as mentioned [here](https://github.com/sharkdp/fd/issues/485#issuecomment-532869261).

Great!



> I'm using `std::env::home_dir()` (which is deprecated) instead of using the `dirs` crate, because this is what the `ignore` crate does. Which would you prefer?

I prefer using the same solution [that we have for `bat`](https://github.com/sharkdp/bat/blob/abeac8b12e9e4c0bc6d9c03ce0d12751a81e9757/src/bin/bat/directories.rs#L6-L8): follow the [XDG standard](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html), i.e. use `$XDG_CONFIG_HOME`, if available. Otherwise, fall back to OS-specific standard directories. On macOS, use `~/.config/fd` as well (not what `dirs` suggests). We could copy some code from `bat`. We don't need the `get_cache_dir` opart.
soedirgo: Thanks for the review!

I changed the global ignore file's behavior so that it gets ignored with `-I`. I think it should behave like `--ignore-file`s, but `-I` should ignore anything that isn't explicit (i.e. `--ignore-file`s).
sharkdp: Thank you for the updates.

Two things:

First, could we please make the documentation less Unix-specific? (i.e. paths like `~/.config/fd/ignore`)

Second, I know this will be hard to add to integration tests. But could you please show a list of tests that you have performed locally to make sure that this works as intended? In particular, I'm thinking about
* Main functionality. Does it work with globs like `*.txt` and absolute paths like `/sys`?
* Prioritization of different ignore files
* What happens if invalid patterns are in the global ignore file?
* Interplay with `--no-ignore` and `--unrestricted`
* Interplay with `--no-ignore-vcs` (files from global ignore file should still be ignored)
soedirgo: > First, could we please make the documentation less Unix-specific? (i.e. paths like ~/.config/fd/ignore)

Updated.

> Second, I know this will be hard to add to integration tests. But could you please show a list of tests that you have performed locally to make sure that this works as intended?

- Functionality is like `--ignore-file`s, i.e. it works with globs, but absolute paths are matched such that the root is the working directory.
- From what I can see, priority is `fd/ignore` < `--ignore-file`s < `.gitignore` < `.ignore` < `.fdignore`.
- Invalid patterns will print an error and ignore the pattern, but other patterns will still be matched.
- `--no-ignore` and `--unrestricted` ignores the global ignore file.
- `--no-ignore-vsc` has no effect.

Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index d419b4f..77e6139 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,7 +2,7 @@
 
 ## Features
 
-
+- Add new `--owner [user][:group]` filter. See #307 (pull #581) (@alexmaco)
 
 ## Bugfixes
 
diff --git a/Cargo.lock b/Cargo.lock
index 20328c7..f605640 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -137,6 +137,7 @@ dependencies = [
  "regex 1.3.6 (registry+https://github.com/rust-lang/crates.io-index)",
  "regex-syntax 0.6.17 (registry+https://github.com/rust-lang/crates.io-index)",
  "tempdir 0.3.7 (registry+https://github.com/rust-lang/crates.io-index)",
+ "users 0.10.0 (registry+https://github.com/rust-lang/crates.io-index)",
  "version_check 0.9.1 (registry+https://github.com/rust-lang/crates.io-index)",
 ]
 
@@ -410,6 +411,15 @@ name = "unicode-width"
 version = "0.1.7"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 
+[[package]]
+name = "users"
+version = "0.10.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+dependencies = [
+ "libc 0.2.69 (registry+https://github.com/rust-lang/crates.io-index)",
+ "log 0.4.8 (registry+https://github.com/rust-lang/crates.io-index)",
+]
+
 [[package]]
 name = "vec_map"
 version = "0.8.1"
@@ -522,6 +532,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 "checksum textwrap 0.11.0 (registry+https://github.com/rust-lang/crates.io-index)" = "d326610f408c7a4eb6f51c37c330e496b08506c9457c9d34287ecc38809fb060"
 "checksum thread_local 1.0.1 (registry+https://github.com/rust-lang/crates.io-index)" = "d40c6d1b69745a6ec6fb1ca717914848da4b44ae29d9b3080cbee91d72a69b14"
 "checksum unicode-width 0.1.7 (registry+https://github.com/rust-lang/crates.io-index)" = "caaa9d531767d1ff2150b9332433f32a24622147e5ebb1f26409d5da67afd479"
+"checksum users 0.10.0 (registry+https://github.com/rust-lang/crates.io-index)" = "aa4227e95324a443c9fcb06e03d4d85e91aabe9a5a02aa818688b6918b6af486"
 "checksum vec_map 0.8.1 (registry+https://github.com/rust-lang/crates.io-index)" = "05c78687fb1a80548ae3250346c3db86a80a7cdd77bda190189f2d0a0987c81a"
 "checksum version_check 0.9.1 (registry+https://github.com/rust-lang/crates.io-index)" = "078775d0255232fb988e6fccf26ddc9d1ac274299aaedcedce21c6f72cc533ce"
 "checksum void 1.0.2 (registry+https://github.com/rust-lang/crates.io-index)" = "6a02e4885ed3bc0f2de90ea6dd45ebcbb66dacffe03547fadbb0eeae2770887d"
diff --git a/Cargo.toml b/Cargo.toml
index 2cc0819..14dec26 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -46,11 +46,15 @@ humantime = "2.0"
 lscolors = "0.7"
 globset = "0.4"
 anyhow = "1.0"
+dirs = "2.0"
 
 [dependencies.clap]
 version = "2.31.2"
 features = ["suggestions", "color", "wrap_help"]
 
+[target.'cfg(unix)'.dependencies]
+users = "0.10.0"
+
 [target.'cfg(all(unix, not(target_os = "redox")))'.dependencies]
 libc = "0.2"
 
diff --git a/README.md b/README.md
index 32e589c..ad7c7bf 100644
--- a/README.md
+++ b/README.md
@@ -458,6 +458,10 @@ To make exclude-patterns like these permanent, you can create a `.fdignore` file
 ```
 Note: `fd` also supports `.ignore` files that are used by other programs such as `rg` or `ag`.
 
+If you want `fd` to ignore these patterns globally, you can put them in `fd`'s global ignore file.
+This is usually located in `~/.config/fd/ignore` in macOS or Linux, and `%APPDATA%\fd\ignore` in
+Windows.
+
 ### Using fd with `xargs` or `parallel`
 
 If we want to run a command on all search results, we can pipe the output to `xargs`:
@@ -554,6 +558,29 @@ After installing `find-file-in-project`, add the line `(setq ffip-use-rust-fd t)
 In emacs, run `M-x find-file-in-project-by-selected` to find matching files. Alternatively, run
 `M-x find-file-in-project` to list all available files in the project.
 
+#### Printing fd's output as a tree
+
+To format the output of `fd` similar to the `tree` command, install [`as-tree`] and pipe the output
+of `fd` to `as-tree`:
+```bash
+fd | as-tree
+```
+
+This can be more useful than running `tree` by itself because `tree` does not ignore any files by
+default, nor does it support as rich a set of options as `fd` does to control what to print:
+```bash
+❯ fd --extension rs | as-tree
+.
+├── build.rs
+└── src
+    ├── app.rs
+    └── error.rs
+```
+
+For more information about `as-tree`, see [the `as-tree` README][`as-tree`].
+
+[`as-tree`]: https://github.com/jez/as-tree
+
 ## License
 Copyright (c) 2017-2020 The fd developers
 
diff --git a/doc/fd.1 b/doc/fd.1
index 3c31cd7..9bc0352 100644
--- a/doc/fd.1
+++ b/doc/fd.1
@@ -33,10 +33,9 @@ Include hidden files and directories in the search results
 .B \-I, \-\-no\-ignore
 Show search results from files and directories that would otherwise be ignored by
 .IR .gitignore ,
-.I .ignore
-or
-.I .fdignore
-files.
+.IR .ignore ,
+.IR .fdignore ,
+or the global ignore file.
 .TP
 .B \-u, \-\-unrestricted
 Alias for '--no-ignore'. Can be repeated; '-uu' is an alias for '--no-ignore --hidden'.
@@ -224,6 +223,15 @@ Examples:
   \-\-changed-before "2018-10-27 10:00:00"
   \-\-change-older-than 2weeks
 .TP
+.BI "-o, \-\-owner " [user][:group]
+Filter files by their user and/or group. Format: [(user|uid)][:(group|gid)]. Either side
+is optional. Precede either side with a '!' to exclude files instead.
+
+Examples:
+  \-\-owner john
+  \-\-owner :students
+  \-\-owner "!john:students"
+.TP
 .BI "\-x, \-\-exec " command
 Execute
 .I command
@@ -273,6 +281,14 @@ The glob syntax is documented here:
 .B LS_COLORS
 Determines how to colorize search results, see
 .BR dircolors (1) .
+.TP
+.B XDG_CONFIG_HOME, HOME
+Used to locate the global ignore file. If
+.B XDG_CONFIG_HOME
+is set, use
+.IR $XDG_CONFIG_HOME/fd/ignore .
+Otherwise, use
+.IR $HOME/.config/fd/ignore .
 .SH EXAMPLES
 .TP
 .RI "Find files and directories that match the pattern '" needle "':"
diff --git a/src/app.rs b/src/app.rs
index 8c23e23..bad5a79 100644
--- a/src/app.rs
+++ b/src/app.rs
@@ -30,7 +30,7 @@ pub fn build_app() -> App<'static, 'static> {
                 .help("Do not respect .(git|fd)ignore files")
                 .long_help(
                     "Show search results from files and directories that would otherwise be \
-                         ignored by '.gitignore', '.ignore' or '.fdignore' files.",
+                         ignored by '.gitignore', '.ignore', '.fdignore', or the global ignore file.",
                 ),
         )
         .arg(
@@ -43,6 +43,12 @@ pub fn build_app() -> App<'static, 'static> {
                          ignored by '.gitignore' files.",
                 ),
         )
+        .arg(
+            Arg::with_name("no-global-ignore-file")
+                .long("no-global-ignore-file")
+                .hidden(true)
+                .long_help("Do not respect the global ignore file."),
+        )
         .arg(
             Arg::with_name("rg-alias-hidden-ignore")
                 .short("u")
@@ -524,6 +530,26 @@ pub fn build_app() -> App<'static, 'static> {
                 ),
         );
 
+    if cfg!(unix) {
+        app = app.arg(
+            Arg::with_name("owner")
+                .long("owner")
+                .short("o")
+                .takes_value(true)
+                .value_name("user:group")
+                .help("Filter by owning user and/or group")
+                .long_help(
+                    "Filter files by their user and/or group. \
+                     Format: [(user|uid)][:(group|gid)]. Either side is optional. \
+                     Precede either side with a '!' to exclude files instead.\n\
+                     Examples:\n    \
+                         --owner john\n    \
+                         --owner :students\n    \
+                         --owner '!john:students'",
+                ),
+        );
+    }
+
     // Make `--one-file-system` available only on Unix and Windows platforms, as per the
     // restrictions on the corresponding option in the `ignore` crate.
     // Provide aliases `mount` and `xdev` for people coming from `find`.
diff --git a/src/filter/mod.rs b/src/filter/mod.rs
index e513c0c..5e45d3b 100644
--- a/src/filter/mod.rs
+++ b/src/filter/mod.rs
@@ -1,5 +1,11 @@
 pub use self::size::SizeFilter;
 pub use self::time::TimeFilter;
 
+#[cfg(unix)]
+pub use self::owner::OwnerFilter;
+
 mod size;
 mod time;
+
+#[cfg(unix)]
+mod owner;
diff --git a/src/filter/owner.rs b/src/filter/owner.rs
new file mode 100644
index 0000000..b69c2ef
--- /dev/null
+++ b/src/filter/owner.rs
@@ -0,0 +1,123 @@
+use anyhow::{anyhow, Result};
+use std::fs;
+
+#[derive(Clone, Copy, Debug, PartialEq)]
+pub struct OwnerFilter {
+    uid: Check<u32>,
+    gid: Check<u32>,
+}
+
+#[derive(Clone, Copy, Debug, PartialEq)]
+enum Check<T> {
+    Equal(T),
+    NotEq(T),
+    Ignore,
+}
+
+impl OwnerFilter {
+    /// Parses an owner constraint
+    /// Returns an error if the string is invalid
+    /// Returns Ok(None) when string is acceptable but a noop (such as "" or ":")
+    pub fn from_string(input: &str) -> Result<Option<Self>> {
+        let mut it = input.split(':');
+        let (fst, snd) = (it.next(), it.next());
+
+        if it.next().is_some() {
+            return Err(anyhow!(
+                "more than one ':' present in owner string '{}'. See 'fd --help'.",
+                input
+            ));
+        }
+
+        let uid = Check::parse(fst, |s| {
+            s.parse()
+                .ok()
+                .or_else(|| users::get_user_by_name(s).map(|user| user.uid()))
+                .ok_or_else(|| anyhow!("'{}' is not a recognized user name", s))
+        })?;
+        let gid = Check::parse(snd, |s| {
+            s.parse()
+                .ok()
+                .or_else(|| users::get_group_by_name(s).map(|group| group.gid()))
+                .ok_or_else(|| anyhow!("'{}' is not a recognized group name", s))
+        })?;
+
+        if let (Check::Ignore, Check::Ignore) = (uid, gid) {
+            Ok(None)
+        } else {
+            Ok(Some(OwnerFilter { uid, gid }))
+        }
+    }
+
+    pub fn matches(&self, md: &fs::Metadata) -> bool {
+        use std::os::unix::fs::MetadataExt;
+
+        self.uid.check(md.uid()) && self.gid.check(md.gid())
+    }
+}
+
+impl<T: PartialEq> Check<T> {
+    fn check(&self, v: T) -> bool {
+        match self {
+            Check::Equal(x) => v == *x,
+            Check::NotEq(x) => v != *x,
+            Check::Ignore => true,
+        }
+    }
+
+    fn parse<F>(s: Option<&str>, f: F) -> Result<Self>
+    where
+        F: Fn(&str) -> Result<T>,
+    {
+        let (s, equality) = match s {
+            Some("") | None => return Ok(Check::Ignore),
+            Some(s) if s.starts_with('!') => (&s[1..], false),
+            Some(s) => (s, true),
+        };
+
+        f(s).map(|x| {
+            if equality {
+                Check::Equal(x)
+            } else {
+                Check::NotEq(x)
+            }
+        })
+    }
+}
+
+#[cfg(test)]
+mod owner_parsing {
+    use super::OwnerFilter;
+
+    macro_rules! owner_tests {
+        ($($name:ident: $value:expr => $result:pat,)*) => {
+            $(
+                #[test]
+                fn $name() {
+                    let o = OwnerFilter::from_string($value);
+                    match o {
+                        $result => {},
+                        _ => panic!("{:?} does not match {}", o, stringify!($result)),
+                    }
+                }
+            )*
+        };
+    }
+
+    use super::Check::*;
+    owner_tests! {
+        empty:      ""      => Ok(None),
+        uid_only:   "5"     => Ok(Some(OwnerFilter { uid: Equal(5), gid: Ignore     })),
+        uid_gid:    "9:3"   => Ok(Some(OwnerFilter { uid: Equal(9), gid: Equal(3)   })),
+        gid_only:   ":8"    => Ok(Some(OwnerFilter { uid: Ignore,   gid: Equal(8)   })),
+        colon_only: ":"     => Ok(None),
+        trailing:   "5:"    => Ok(Some(OwnerFilter { uid: Equal(5), gid: Ignore     })),
+
+        uid_negate: "!5"    => Ok(Some(OwnerFilter { uid: NotEq(5), gid: Ignore     })),
+        both_negate:"!4:!3" => Ok(Some(OwnerFilter { uid: NotEq(4), gid: NotEq(3)   })),
+        uid_not_gid:"6:!8"  => Ok(Some(OwnerFilter { uid: Equal(6), gid: NotEq(8)   })),
+
+        more_colons:"3:5:"  => Err(_),
+        only_colons:"::"    => Err(_),
+    }
+}
diff --git a/src/filter/size.rs b/src/filter/size.rs
index 895169e..08f6c68 100644
--- a/src/filter/size.rs
+++ b/src/filter/size.rs
@@ -2,7 +2,7 @@ use lazy_static::lazy_static;
 use regex::Regex;
 
 lazy_static! {
-    static ref SIZE_CAPTURES: Regex = { Regex::new(r"(?i)^([+-])(\d+)(b|[kmgt]i?b?)$").unwrap() };
+    static ref SIZE_CAPTURES: Regex = Regex::new(r"(?i)^([+-])(\d+)(b|[kmgt]i?b?)$").unwrap();
 }
 
 #[derive(Clone, Copy, Debug, PartialEq)]
diff --git a/src/main.rs b/src/main.rs
index 377a698..fa0e462 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -25,6 +25,8 @@ use regex::bytes::{RegexBuilder, RegexSetBuilder};
 use crate::exec::CommandTemplate;
 use crate::exit_codes::ExitCode;
 use crate::filetypes::FileTypes;
+#[cfg(unix)]
+use crate::filter::OwnerFilter;
 use crate::filter::{SizeFilter, TimeFilter};
 use crate::options::Options;
 use crate::regex_helper::pattern_has_uppercase_char;
@@ -275,6 +277,13 @@ fn run() -> Result<ExitCode> {
         }
     }
 
+    #[cfg(unix)]
+    let owner_constraint = if let Some(s) = matches.value_of("owner") {
+        OwnerFilter::from_string(s)?
+    } else {
+        None
+    };
+
     let config = Options {
         case_sensitive,
         search_full_path: matches.is_present("full-path"),
@@ -285,6 +294,9 @@ fn run() -> Result<ExitCode> {
         read_vcsignore: !(matches.is_present("no-ignore")
             || matches.is_present("rg-alias-hidden-ignore")
             || matches.is_present("no-ignore-vcs")),
+        read_global_ignore: !(matches.is_present("no-ignore")
+            || matches.is_present("rg-alias-hidden-ignore")
+            || matches.is_present("no-global-ignore-file")),
         follow_links: matches.is_present("follow"),
         one_file_system: matches.is_present("one-file-system"),
         null_separator: matches.is_present("null_separator"),
@@ -358,6 +370,8 @@ fn run() -> Result<ExitCode> {
             .unwrap_or_else(|| vec![]),
         size_constraints: size_limits,
         time_constraints,
+        #[cfg(unix)]
+        owner_constraint,
         show_filesystem_errors: matches.is_present("show-errors"),
         path_separator,
         max_results: matches
diff --git a/src/options.rs b/src/options.rs
index 3f52516..dd11506 100644
--- a/src/options.rs
+++ b/src/options.rs
@@ -5,6 +5,8 @@ use regex::bytes::RegexSet;
 
 use crate::exec::CommandTemplate;
 use crate::filetypes::FileTypes;
+#[cfg(unix)]
+use crate::filter::OwnerFilter;
 use crate::filter::{SizeFilter, TimeFilter};
 
 /// Configuration options for *fd*.
@@ -25,6 +27,9 @@ pub struct Options {
     /// Whether to respect VCS ignore files (`.gitignore`, ..) or not.
     pub read_vcsignore: bool,
 
+    /// Whether to respect the global ignore file or not.
+    pub read_global_ignore: bool,
+
     /// Whether to follow symlinks or not.
     pub follow_links: bool,
 
@@ -82,6 +87,10 @@ pub struct Options {
     /// Constraints on last modification time of files
     pub time_constraints: Vec<TimeFilter>,
 
+    #[cfg(unix)]
+    /// User/group ownership constraint
+    pub owner_constraint: Option<OwnerFilter>,
+
     /// Whether or not to display filesystem errors
     pub show_filesystem_errors: bool,
 
diff --git a/src/walk.rs b/src/walk.rs
index 93f20de..1082e1c 100644
--- a/src/walk.rs
+++ b/src/walk.rs
@@ -1,4 +1,5 @@
 use std::borrow::Cow;
+use std::env;
 use std::ffi::OsStr;
 use std::fs::{FileType, Metadata};
 use std::io;
@@ -82,6 +83,34 @@ pub fn scan(path_vec: &[PathBuf], pattern: Arc<Regex>, config: Arc<Options>) ->
         walker.add_custom_ignore_filename(".fdignore");
     }
 
+    if config.read_global_ignore {
+        #[cfg(target_os = "macos")]
+        let config_dir_op = env::var_os("XDG_CONFIG_HOME")
+            .map(PathBuf::from)
+            .filter(|p| p.is_absolute())
+            .or_else(|| dirs::home_dir().map(|d| d.join(".config")));
+
+        #[cfg(not(target_os = "macos"))]
+        let config_dir_op = dirs::config_dir();
+
+        if let Some(global_ignore_file) = config_dir_op
+            .map(|p| p.join("fd").join("ignore"))
+            .filter(|p| p.is_file())
+        {
+            let result = walker.add_ignore(global_ignore_file);
+            match result {
+                Some(ignore::Error::Partial(_)) => (),
+                Some(err) => {
+                    print_error(format!(
+                        "Malformed pattern in global ignore file. {}.",
+                        err.to_string()
+                    ));
+                }
+                None => (),
+            }
+        }
+    }
+
     for ignore_file in &config.ignore_files {
         let result = walker.add_ignore(ignore_file);
         match result {
@@ -411,6 +440,19 @@ fn spawn_senders(
                 }
             }
 
+            #[cfg(unix)]
+            {
+                if let Some(ref owner_constraint) = config.owner_constraint {
+                    if let Ok(ref metadata) = entry_path.metadata() {
+                        if !owner_constraint.matches(&metadata) {
+                            return ignore::WalkState::Continue;
+                        }
+                    } else {
+                        return ignore::WalkState::Continue;
+                    }
+                }
+            }
+
             // Filter out unwanted sizes if it is a file and we have been given size constraints.
             if !config.size_constraints.is_empty() {
                 if entry_path.is_file() {


Test patch:
diff --git a/tests/testenv/mod.rs b/tests/testenv/mod.rs
index a4414a8..ae3e08e 100644
--- a/tests/testenv/mod.rs
+++ b/tests/testenv/mod.rs
@@ -190,7 +190,7 @@ impl TestEnv {
         // Setup *fd* command.
         let mut cmd = process::Command::new(&self.fd_exe);
         cmd.current_dir(self.temp_dir.path().join(path));
-        cmd.args(args);
+        cmd.arg("--no-global-ignore-file").args(args);
 
         // Run *fd*.
         let output = cmd.output().expect("fd output");
@@ -251,7 +251,7 @@ impl TestEnv {
         // Setup *fd* command.
         let mut cmd = process::Command::new(&self.fd_exe);
         cmd.current_dir(self.temp_dir.path().join(path));
-        cmd.args(args);
+        cmd.arg("--no-global-ignore-file").args(args);
 
         // Run *fd*.
         let output = cmd.output().expect("fd output");