opened
saintger
username_0
Files copied with --reflink=auto on BTRFS are already sharing the same data.
However fdupes identify them as duplicates (and in a technical sense, they are indeed duplicates).
It would be nice if we could have an option to ignore such files.

Thanks for this useful software.
Feature request: ignore files copied with --reflink=auto on BTRFS
issue
created
jbruchon
username_1
How do you detect that a file pair is set in BTRFS to be copy-on-write in C? Unless the inode and device numbers are the same (meaning they're hard linked) it seems like it'd take some btrfs-specific code to detect.
comment
created
saintger
username_0
I am not expert, but here is someone else who has the same question and got some answers:
https://www.spinics.net/lists/linux-btrfs/msg19075.html
I have no idea if the code would be specific to BTRFS or applicable to CoW filesystems in general.
comment
created
jbruchon
username_1
It appears that the code would need to be Linux-specific, using FIEMAP: https://www.mjmwired.net/kernel/Documentation/filesystems/fiemap.txt

Relevant post in the list you referenced: https://www.spinics.net/lists/linux-btrfs/msg19114.html
comment
created
saintger
username_0
Indeed, thanks for having looked.
I just saw your jdupes fork, looks interesting !
The reason for this feature request, was to use fdupes with duperemove, in order to use BTRFS deduplication.
However it seems that I can just use your jdupes for that.
Is your "--dedupe" option is considered stable ?
I have not seen anyone else using it.
comment
created
jbruchon
username_1
User @yath here posted the code for that feature here: https://github.com/adrianlopezroche/fdupes/pull/52/commits/b9913d847af7570c7e46cf2e70d93ea7d0f30651 which resulted in this commit: https://github.com/username_1/jdupes/commit/80dedfc86d4b883a6980e9a72fa2c36c4c164eae

I simply worked with yath to integrate it into my code base since it seemed like some people would find it useful. I don't know who else is using it other than yath because I don't use btrfs in the first place. All it does is pass the file pairs to the kernel's btrfs driver for low-level deduplication, and the kernel verifies that the file contents are identical before deduplicating anyway. I'd say that the feature is stable and the kernel guarantees that the feature can't do anything more dangerous than a btrfs dedupe of two files.

One of the things I did differently was to not do byte-for-byte comparison before passing the files to the kernel for dedupe; since the kernel does that itself anyway, it's a lot of wasted time, so I used a "quick compare" semi-documented flag I already had written to skip that phase for better performance.

Your post was interesting to me because it could be useful; files that are btrfs deduped already can essentially be treated as hard links for the purpose of duplicate scanning, but comparing inode/dev pairs alone don't reveal that kind of data relationship. Since btrfs is fairly popular I'd be interested in adding the feature at some point for faster operation.
comment
created
saintger
username_0
For info, rmlint has tried to implement this feature:
https://github.com/sahib/rmlint/issues/132
https://github.com/sahib/rmlint/blob/develop/lib/formats/sh.c.in#L89-L96
comment