Repository: sharkdp/fd (language: rust)

Issue:
Without pattern 'find' ~8x faster than 'fd'
OS: Linux 4.14.3-1-ARCH
FS: ext4
fd: 6.0.0

Simple test:
```bash
> time find > /dev/null
find > /dev/null  0.00s user 0.06s system 97% cpu 0.065 total
> time fd --color never -HI > /dev/null
fd --color never -HI > /dev/null  0.75s user 0.10s system 133% cpu 0.631 total
```

Also `fd` with pattern faster ~8x than `fd` without pattern:
```bash
> time fd --color never -HI '.zshrc' > /dev/null
fd --color never -HI '.zshrc' > /dev/null  0.18s user 0.06s system 325% cpu 0.073 total
> time fd --color never -HI > /dev/null
fd --color never -HI > /dev/null  0.74s user 0.06s system 132% cpu 0.608 total
```


Discussion / hints:
OS: Debian testing
FS: ext4
fd: 6.0.0

Installed fd and tried it on my camera folder. The results are here:

`find . -iname '*.jpg'  0.08s user 0.06s system 66% cpu 0.210 total`

`fd '.*\.jpg$' .  0.41s user 0.16s system 95% cpu 0.602 total`

The time taken and cpu usage are very high contrary to the expectation.
Interesting, thank you for reporting this.

How many entries are found (without the search pattern)?

Also, how many cores does your machine have?
Directory size: 27G
Count of entries: 35809
Count of cores: 4
Machine details:
```
System:    Host: brahman Kernel: 4.14.0-2.3-liquorix-amd64 x86_64 bits: 64 Desktop: Openbox 3.6.1
           Distro: Debian GNU/Linux buster/sid
Machine:   Device: desktop Mobo: ASUSTeK model: M4A78LT-M-LE v: Rev X.0x serial: N/A
           BIOS: American Megatrends v: 0801 date: 01/17/2011
CPU:       Quad core AMD Phenom II X4 960T (-MCP-) cache: 2048 KB
           clock speeds: max: 3000 MHz 1: 1600 MHz 2: 3000 MHz 3: 800 MHz 4: 3000 MHz
```
Directory size: 261 GiB
Number of files: 
```
> find . -name "*.jpg" | wc -l
16049
> fd ".*\.jpg$" . | wc -l     
17006
> find . -iname "*.jpg" | wc -l
17061
> find . -iregex ".*\.jpg$" | wc -l
17061
```
Even the numbers reported are different.
@dtcyganok Thanks. Are you using the *fd* version from the official Arch repositories? Or did you build it yourself?

@alaymari 
> Even the numbers reported are different.

Please make sure to run `fd` with the `-I` and `-H` options. Otherwise you can not expect the results to be the same. Also, just as a remark, you can use `fd -e jpg` to perform searches for a particular file extension. So you could try:
```
> fd -HI -e jpg | wc -l
```
I am using the `fd` from the official Arch repositories
@sharkdp 
```
> fd -HI -e jpg | wc -l
17061
```
No change :-) This is just FYI. 

Hope you will be coming out with a solution for this problem soon.
So fd is slower than find when you are not using any pattern? Anyway to avoid that?
I might have an idea what's causing this.

@dtcyganok Could you please run your initial benchmark and add `--max-buffer-time 0` as an additional option?

Fix patch:
diff --git a/src/internal.rs b/src/internal.rs
index 88fd1c710..c694045e0 100644
--- a/src/internal.rs
+++ b/src/internal.rs
@@ -101,6 +101,9 @@ fn expr_has_uppercase_char(expr: &Expr) -> bool {
     }
 }
 
+/// Maximum size of the output buffer before flushing results to the console
+pub const MAX_BUFFER_LENGTH: usize = 1000;
+
 /// Exit code representing a general error
 pub const EXITCODE_ERROR: i32 = 1;
 
diff --git a/src/walk.rs b/src/walk.rs
index 584fd5e20..f30226acd 100644
--- a/src/walk.rs
+++ b/src/walk.rs
@@ -10,7 +10,7 @@ extern crate ctrlc;
 
 use exec;
 use fshelper;
-use internal::{error, FdOptions, EXITCODE_SIGINT};
+use internal::{error, FdOptions, EXITCODE_SIGINT, MAX_BUFFER_LENGTH};
 use output;
 
 use std::process;
@@ -147,8 +147,10 @@ pub fn scan(path_vec: &[PathBuf], pattern: Arc<Regex>, config: Arc<FdOptions>) {
                     ReceiverMode::Buffering => {
                         buffer.push(value);
 
-                        // Have we reached the maximum time?
-                        if time::Instant::now() - start > max_buffer_time {
+                        // Have we reached the maximum buffer size or maximum buffering time?
+                        if buffer.len() > MAX_BUFFER_LENGTH
+                            || time::Instant::now() - start > max_buffer_time
+                        {
                             // Flush the buffer
                             for v in &buffer {
                                 output::print_entry(v, &rx_config, &receiver_wtq);


Test patch: