Repository: sharkdp/fd (language: rust)

Issue:
fd can get stuck when ran onto the whole FS from the root
The ran command is just `fd foobar /`. It gets stuck and never end. I sadly was unable to understand what's causing it.
I can investigate to give answers if you have some ideas about what else could I check.

Below is the partition's setup:

```
# lsblk
NAME           MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda              8:0    0 223,6G  0 disk  
├─sda1           8:1    0   512M  0 part  /boot/efi
├─sda2           8:2    0   200M  0 part  
│ └─cryptboot  254:3    0   198M  0 crypt /boot
└─sda3           8:3    0 222,9G  0 part  
  └─lvm        254:0    0 222,9G  0 crypt 
    ├─vg0-swap 254:1    0     8G  0 lvm   [SWAP]
    └─vg0-root 254:2    0 214,9G  0 lvm   /
```

EDIT: Issue is reproducing everytime on this Archlinux setup. Couldn't reproduce it on another Archlinux with unencrypted system.
fd can get stuck when ran onto the whole FS from the root
The ran command is just `fd foobar /`. It gets stuck and never end. I sadly was unable to understand what's causing it.
I can investigate to give answers if you have some ideas about what else could I check.

Below is the partition's setup:

```
# lsblk
NAME           MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda              8:0    0 223,6G  0 disk  
├─sda1           8:1    0   512M  0 part  /boot/efi
├─sda2           8:2    0   200M  0 part  
│ └─cryptboot  254:3    0   198M  0 crypt /boot
└─sda3           8:3    0 222,9G  0 part  
  └─lvm        254:0    0 222,9G  0 crypt 
    ├─vg0-swap 254:1    0     8G  0 lvm   [SWAP]
    └─vg0-root 254:2    0 214,9G  0 lvm   /
```

EDIT: Issue is reproducing everytime on this Archlinux setup. Couldn't reproduce it on another Archlinux with unencrypted system.


Discussion / hints:
I can reproduce this on my PC. This seems to be a permission problem with files in `/proc` and `/sys`.

Could you try to run
```
fd -E /proc -E /sys foobar /
```
where `-E` (`--exclude`) excludes these two folders?

(Note: you might have to exclude some other mount points as well)
Indeed this does work. Strange that my personal server doesn't have the same issue though.
Oh, and the issue was present even when running fd as root. So I'm not sure about the permission idea.
> Oh, and the issue was present even when running fd as root. So I'm not sure about the permission idea.

I see, interesting. Something else must be going on inside `/proc` and/or `/sys`. If anybody has an idea on how to fix this, I'd be glad for any hints.
Excluding only `proc` is enough, so problem comes from something inside `/proc`.
343 items inside `/proc` makes a lot of things to test though.
Even more interesting.
`fd -E "/proc/[0-9]*" foobar /` is succeeding.
`fd -E "/proc/[0-9]*" foobar /proc` is failing.
Wait, this also seems to change from time to time. Right now, `fd foobar /` works perfectly fine on my machine(?!?).
I suspect this could be caused by infinite recursion within `/proc` (see [here](https://stackoverflow.com/q/4740818/704831) or [here](https://stackoverflow.com/q/20914606/704831)).

Quote:
> Avoiding symbolic links may not be sufficient for avoiding "infinite" recursion in /proc. On the bright side, "infinite" is bounded by PATH_MAX... 

Unfortunately, I cannot reproduce the problem at the moment. Could you try
```
fd --max-depth 50 foobar /
```
?
I'm going to close this (for now), as there is no further feedback. Feel free to comment here if this should be re-opened.

The workaround for now is to explicitly exclude `/proc` via

```
fd -E /proc ...
```
Sorry, didn't knew you was waiting for input: it was added with an edit, so I wasn't notified: original comment was not asking for input. Please post a new comment in such situation.

I tested what you asked for: didn't solved the issue, so I guess this isn't infinite recursion.
And while it hang like that, CPU is using every single core at 100%.

I guess we could then reopen it?

EDIT: Even if I'm the opener I can't reopen myself.
By the way, this was on Linux computers.
I'm currently running it on an old mac. I guess it's gonna take long (old, bad shaped HDD, no SSD in it), so I'll let it run, but I think that when I'll be back this evening computer will be brozen with out of memory: currently running for like 10 minutes, CPU is only around 10%, but memory constantly increase. It started low and currently is already at 140MB.

EDIT: CPU lowered to 1.5-2%, but it still continue.

EDIT2: Might not be the bug on the mac, 30 minutes later, it found a file with "foobar" in the name (!) at the 9th level of depth. So I'll see this evening if it ended correctly, the HDD is kinda slow, and probably fd is slowed down by I/O on disk rather than CPU like other SSD-using computers.
However, is it normal for the memory to constantly increase over the time?
So, not reproduced on the mac, it just took quite some time.
Thank you for investigating.

To summarize (correct me if I'm wrong):

- `fd` can get stuck in an infinite (or very long?) loop when searching `/`.
- This seems to be caused by something weird going on in `/proc`.
- The problem is not 100% reproducible (I'm pretty sure I saw this on my Linux machine, but I can not reproduce it now - nor on three other Linux machines).
- The problem can be prevented by excluding `/proc` or `/proc/[0-9]*`.
Well, that's pretty much it except that I detected a weird case in comment https://github.com/sharkdp/fd/issues/288#issuecomment-383637131
It looks like excluding `/proc/[0-9]*` and searching in `/` make it work but excluding the same thing and searching in `/proc`make it fail… which is pure nonsense to me. I was trying the find if there was a culprit file and stopped when I discovered that because that didn't make any sense.

The exact same computer I did the tests on yesterday morning now succeed on `fd foobar /`… I'll try to test the other one that had issue when coming to work.
Yeah, this seems really random. The laptop now do it again, the one that didn't 2H ago and the desktop don't anymore…
On the laptop it spawns 10 threads, and currently 4 of them are running a core at 100%. I've seen all my threads at 100% previous week.

Do you have any idea how to debug that? See where does it hang like that?

And this is kinda out of topic, but per comment https://github.com/sharkdp/fd/issues/288#issuecomment-386208380 is it normal that on the mac case memory did increase during the whole search process?
> Do you have any idea how to debug that? See where does it hang like that?

Maybe `fd . /`?

> And this is kinda out of topic, but per comment #288 (comment) is it normal that on the mac case memory did increase during the whole search process?

I'm not sure, I will look into it.
Okay, so running `fd . /` on the desktop computer reliably fail on `/proc/11364/task/11364/net
`.
That process is a Firefox one: `clement  11364  0.0  0.0      0     0 ?        Z    10:22   0:00 [firefox] <defunct>`

The laptop's case however is more complicated. It seems to stop randomly on any file at some point. I can share some of them here if you think that's useful but I'm not sure about that. Maybe a doing single thread might help. Don't know why one computer is affected while the other isn't though.
Okay, actually I just RTFM'd a bit and used `-j 1`.

Got stuck on `/proc/19848/net`

`ps aux` give that:
`clement  19848  0.0  0.0      0     0 ?        Z    08:46   0:00 [pingsender] <defunct>`
Oh, and surprisingly, using `-j 1` still shows two processes. Both use 100% of the CPU.

And last thing but I've no idea about: sometimes when the bug triggers it "breaks" the terminal I used, terminology. ie. display was still refreshed, but I couldn't type anymore, even when I opened new windows. Only happened one time or two, and got fixed when I pkill'd the process from elsewhere.

EDIT: When it happens, ctrl + c don't do the trick, but seems to work if you wait a long time and retry after having done so a first time.
Thank you very much for your investigation!

It looks like you are up to something with the `<defunct>` processes! I can now reliably reproduce this by creating a zombie process on purpose:

1. Run `fd foobar /proc` => everything is fine
2. Copy the code from https://stackoverflow.com/a/25228579/704831 into a file called `zombie.c`
3. Compile it: `gcc -o zombie zombie.c`.
4. Create the zombie process: `./zombie`.
5. Run `ps -ef | grep defunct` in a new terminal. It should show `[zombie] <defunct>`.
6. Run `fd foobar /proc` while `zombie` is still running. It will hang.
7. Stop `zombie` and run `fd foobar /proc` again => everything is fine.

Experiment 2:

1. Run `fd foobar /` => everything ok
2. Create the zombie process: `./zombie`.
3. Get the PID of the defunct process via `ps -ef | grep defunct`
4. Call `fd foobar / -E /proc/<PID>` => everything ok
5. Call `fd foobar /` => it hangs
(see the linked ticked in ripgrep for some further debugging)
This might be a bug in the `ReadDir` iterator in Rusts standard library. The issue has been reported here: https://github.com/rust-lang/rust/issues/50619
My pull request which fixes a bug in Rusts standard library has been merged (https://github.com/rust-lang/rust/pull/50630). Now we have to wait for the next Rust release in order to fix this bug in `fd`.
Actually, we have to wait for Rust 1.29. I forgot about the beta stage. This bug is fixed when compiling `fd` with the current `rustc 1.29.0-beta.1`.
I can reproduce this on my PC. This seems to be a permission problem with files in `/proc` and `/sys`.

Could you try to run
```
fd -E /proc -E /sys foobar /
```
where `-E` (`--exclude`) excludes these two folders?

(Note: you might have to exclude some other mount points as well)
Indeed this does work. Strange that my personal server doesn't have the same issue though.
Oh, and the issue was present even when running fd as root. So I'm not sure about the permission idea.
> Oh, and the issue was present even when running fd as root. So I'm not sure about the permission idea.

I see, interesting. Something else must be going on inside `/proc` and/or `/sys`. If anybody has an idea on how to fix this, I'd be glad for any hints.
Excluding only `proc` is enough, so problem comes from something inside `/proc`.
343 items inside `/proc` makes a lot of things to test though.
Even more interesting.
`fd -E "/proc/[0-9]*" foobar /` is succeeding.
`fd -E "/proc/[0-9]*" foobar /proc` is failing.
Wait, this also seems to change from time to time. Right now, `fd foobar /` works perfectly fine on my machine(?!?).
I suspect this could be caused by infinite recursion within `/proc` (see [here](https://stackoverflow.com/q/4740818/704831) or [here](https://stackoverflow.com/q/20914606/704831)).

Quote:
> Avoiding symbolic links may not be sufficient for avoiding "infinite" recursion in /proc. On the bright side, "infinite" is bounded by PATH_MAX... 

Unfortunately, I cannot reproduce the problem at the moment. Could you try
```
fd --max-depth 50 foobar /
```
?
I'm going to close this (for now), as there is no further feedback. Feel free to comment here if this should be re-opened.

The workaround for now is to explicitly exclude `/proc` via

```
fd -E /proc ...
```
Sorry, didn't knew you was waiting for input: it was added with an edit, so I wasn't notified: original comment was not asking for input. Please post a new comment in such situation.

I tested what you asked for: didn't solved the issue, so I guess this isn't infinite recursion.
And while it hang like that, CPU is using every single core at 100%.

I guess we could then reopen it?

EDIT: Even if I'm the opener I can't reopen myself.
By the way, this was on Linux computers.
I'm currently running it on an old mac. I guess it's gonna take long (old, bad shaped HDD, no SSD in it), so I'll let it run, but I think that when I'll be back this evening computer will be brozen with out of memory: currently running for like 10 minutes, CPU is only around 10%, but memory constantly increase. It started low and currently is already at 140MB.

EDIT: CPU lowered to 1.5-2%, but it still continue.

EDIT2: Might not be the bug on the mac, 30 minutes later, it found a file with "foobar" in the name (!) at the 9th level of depth. So I'll see this evening if it ended correctly, the HDD is kinda slow, and probably fd is slowed down by I/O on disk rather than CPU like other SSD-using computers.
However, is it normal for the memory to constantly increase over the time?
So, not reproduced on the mac, it just took quite some time.
Thank you for investigating.

To summarize (correct me if I'm wrong):

- `fd` can get stuck in an infinite (or very long?) loop when searching `/`.
- This seems to be caused by something weird going on in `/proc`.
- The problem is not 100% reproducible (I'm pretty sure I saw this on my Linux machine, but I can not reproduce it now - nor on three other Linux machines).
- The problem can be prevented by excluding `/proc` or `/proc/[0-9]*`.
Well, that's pretty much it except that I detected a weird case in comment https://github.com/sharkdp/fd/issues/288#issuecomment-383637131
It looks like excluding `/proc/[0-9]*` and searching in `/` make it work but excluding the same thing and searching in `/proc`make it fail… which is pure nonsense to me. I was trying the find if there was a culprit file and stopped when I discovered that because that didn't make any sense.

The exact same computer I did the tests on yesterday morning now succeed on `fd foobar /`… I'll try to test the other one that had issue when coming to work.
Yeah, this seems really random. The laptop now do it again, the one that didn't 2H ago and the desktop don't anymore…
On the laptop it spawns 10 threads, and currently 4 of them are running a core at 100%. I've seen all my threads at 100% previous week.

Do you have any idea how to debug that? See where does it hang like that?

And this is kinda out of topic, but per comment https://github.com/sharkdp/fd/issues/288#issuecomment-386208380 is it normal that on the mac case memory did increase during the whole search process?
> Do you have any idea how to debug that? See where does it hang like that?

Maybe `fd . /`?

> And this is kinda out of topic, but per comment #288 (comment) is it normal that on the mac case memory did increase during the whole search process?

I'm not sure, I will look into it.
Okay, so running `fd . /` on the desktop computer reliably fail on `/proc/11364/task/11364/net
`.
That process is a Firefox one: `clement  11364  0.0  0.0      0     0 ?        Z    10:22   0:00 [firefox] <defunct>`

The laptop's case however is more complicated. It seems to stop randomly on any file at some point. I can share some of them here if you think that's useful but I'm not sure about that. Maybe a doing single thread might help. Don't know why one computer is affected while the other isn't though.
Okay, actually I just RTFM'd a bit and used `-j 1`.

Got stuck on `/proc/19848/net`

`ps aux` give that:
`clement  19848  0.0  0.0      0     0 ?        Z    08:46   0:00 [pingsender] <defunct>`
Oh, and surprisingly, using `-j 1` still shows two processes. Both use 100% of the CPU.

And last thing but I've no idea about: sometimes when the bug triggers it "breaks" the terminal I used, terminology. ie. display was still refreshed, but I couldn't type anymore, even when I opened new windows. Only happened one time or two, and got fixed when I pkill'd the process from elsewhere.

EDIT: When it happens, ctrl + c don't do the trick, but seems to work if you wait a long time and retry after having done so a first time.
Thank you very much for your investigation!

It looks like you are up to something with the `<defunct>` processes! I can now reliably reproduce this by creating a zombie process on purpose:

1. Run `fd foobar /proc` => everything is fine
2. Copy the code from https://stackoverflow.com/a/25228579/704831 into a file called `zombie.c`
3. Compile it: `gcc -o zombie zombie.c`.
4. Create the zombie process: `./zombie`.
5. Run `ps -ef | grep defunct` in a new terminal. It should show `[zombie] <defunct>`.
6. Run `fd foobar /proc` while `zombie` is still running. It will hang.
7. Stop `zombie` and run `fd foobar /proc` again => everything is fine.

Experiment 2:

1. Run `fd foobar /` => everything ok
2. Create the zombie process: `./zombie`.
3. Get the PID of the defunct process via `ps -ef | grep defunct`
4. Call `fd foobar / -E /proc/<PID>` => everything ok
5. Call `fd foobar /` => it hangs
(see the linked ticked in ripgrep for some further debugging)
This might be a bug in the `ReadDir` iterator in Rusts standard library. The issue has been reported here: https://github.com/rust-lang/rust/issues/50619
My pull request which fixes a bug in Rusts standard library has been merged (https://github.com/rust-lang/rust/pull/50630). Now we have to wait for the next Rust release in order to fix this bug in `fd`.
Actually, we have to wait for Rust 1.29. I forgot about the beta stage. This bug is fixed when compiling `fd` with the current `rustc 1.29.0-beta.1`.

Fix patch:
diff --git a/.travis.yml b/.travis.yml
index c365552d2..f43e72e54 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -64,19 +64,19 @@ matrix:
 
     # Minimum Rust supported channel.
     - os: linux
-      rust: 1.20.0
+      rust: 1.29.0
       env: TARGET=x86_64-unknown-linux-gnu
     - os: linux
-      rust: 1.20.0
+      rust: 1.29.0
       env: TARGET=x86_64-unknown-linux-musl
     - os: linux
-      rust: 1.20.0
+      rust: 1.29.0
       env: TARGET=i686-unknown-linux-gnu
     - os: linux
-      rust: 1.20.0
+      rust: 1.29.0
       env: TARGET=i686-unknown-linux-musl
     - os: osx
-      rust: 1.20.0
+      rust: 1.29.0
       env: TARGET=x86_64-apple-darwin
 
     # Code formatting check
diff --git a/README.md b/README.md
index c5e93a856..55f8319f4 100644
--- a/README.md
+++ b/README.md
@@ -218,7 +218,7 @@ With Rust's package manager [cargo](https://github.com/rust-lang/cargo), you can
 ```
 cargo install fd-find
 ```
-Note that rust version *1.20.0* or later is required.
+Note that rust version *1.29.0* or later is required.
 
 ### From binaries
 
diff --git a/build.rs b/build.rs
index 5bc9b4b7a..d45d98b95 100644
--- a/build.rs
+++ b/build.rs
@@ -18,12 +18,11 @@ use std::process::exit;
 include!("src/app.rs");
 
 fn main() {
-    match version_check::is_min_version("1.20") {
-        // rustc >= 1.20
+    match version_check::is_min_version("1.29") {
         Some((true, _)) => {}
-        // rustc < 1.20 or can't figure it out
+        // rustc version too small or can't figure it out
         _ => {
-            writeln!(&mut io::stderr(), "This crate requires rustc >= 1.20").unwrap();
+            writeln!(&mut io::stderr(), "'fd' requires rustc >= 1.29").unwrap();
             exit(1);
         }
     }


Test patch: