Repository: qarmin/czkawka (language: rust)

Issue:
Searching for dupes, then searching again, changes the data shown and is confusing (with caching on)
**Desktop**
 - Czkawka version: 6.0.0 gui
 - OS version: Mac OS 13.5.2
 - hardware: MacBook Pro 2021 (apple silicon)
 - compile and run method: as per instructions, having previously done `brew install rustup`

**Bug Description**

Ensure caching is on.

Create a folder "reference" with some test data (images or whatever), with no dupes in it. Copy the folder to another called "other".

Add the two folders into czkawka. Select "Reference" checkbox next to the "reference" folder.

Do a duplicate file search and then "Select -> Select all". The results look sensible: there's a ref folder listed above every dupe in a non-reference folder.

<img width="1098" alt="Screenshot 2023-10-12 at 12 31 43" src="https://github.com/qarmin/czkawka/assets/793071/e0134fb5-75a6-4b9c-8f09-ada21862d220">

Now click search again, and the display changes: there is only one reference folder file listed, and a bunch of matches in the other folder. Do "Select -> Select all". It will select everything except the one reference folder file.

<img width="1102" alt="Screenshot 2023-10-12 at 12 30 51" src="https://github.com/qarmin/czkawka/assets/793071/26c9490b-d5f3-403d-a520-b5c79826e115">

This is confusing; it now looks to me that this means hitting 'delete' will delete original data, because the 'other' folder matches are non-matching files amongst themselves, which are all now ticked. In fact, I don't think original data *is* going to be deleted; it's just that some dupes in the reference folder are now missing from the display.

Basically, hitting search should be idempotent: the result shouldn't change if you haven't changed anything since the last search.

When I turn caching off and delete existing cache files, I don't get this issue.

P.S. another issue, which I haven't go repro steps for but may be related to the issue above: I've search for dupes than deleted them, then searched again immediately with all same settings and more duplicates come up. I've seen this 2-3 times now.


Discussion / hints:
Can the cli version do "reference folders" btw, or is that just in the gui?
Reference folders is a core feature, so is theoretically available both for gui and cli, but for now only gui have interface to use it.
Are there tests (e.g. unit) for depuplicating functionality, caching etc btw? Had a quick look and couldn't see anything
For now tests are quite basic:
- https://github.com/qarmin/czkawka/blob/9b57382e39d81eabc934fb23aebd26908fb47af8/.github/workflows/linux_cli.yml#L43-L49
- https://github.com/qarmin/czkawka/blob/master/czkawka_core/src/duplicate.rs#L1236
but slowly starts to be more advanced (e.g. with https://github.com/qarmin/czkawka/pull/1083)

Which exactly version did you checked?
Official 6.0.0 or custom build?

I found problem with latest master version(unreleased 6.1.0 version), but I remember that I already touched this code, so it could be recent regression.

Fix patch:
diff --git a/Changelog.md b/Changelog.md
index b88306157..7ebc7de62 100644
--- a/Changelog.md
+++ b/Changelog.md
@@ -1,9 +1,17 @@
 ## Version 6.1.0 - ?
-- BREAKING CHANGE - Changed cache saving method, deduplicated, optimized and simplified procedure - [#1072](https://github.com/qarmin/czkawka/pull/1072)
+- BREAKING CHANGE - Changed cache saving method, deduplicated, optimized and simplified procedure(all files needs to be hashed again) - [#1072](https://github.com/qarmin/czkawka/pull/1072)
 - Remove up to 170ms of delay after ending scan - [#1070](https://github.com/qarmin/czkawka/pull/1070)
 - Added logger with useful info when debugging app (level can be adjusted via e.g. `RUST_LOG=debug` env) - [#1072](https://github.com/qarmin/czkawka/pull/1072), [#1070](https://github.com/qarmin/czkawka/pull/1070)
-- Core code cleanup - [#1072](https://github.com/qarmin/czkawka/pull/1072), [#1070](https://github.com/qarmin/czkawka/pull/1070)
+- Core code cleanup - [#1072](https://github.com/qarmin/czkawka/pull/1072), [#1070](https://github.com/qarmin/czkawka/pull/1070), [#1082](https://github.com/qarmin/czkawka/pull/1082)
 - Updated list of bad extensions and support for finding invalid jar files - [#1070](https://github.com/qarmin/czkawka/pull/1070)
+- More default excluded items on Windows(like pagefile) - [#1074](https://github.com/qarmin/czkawka/pull/1074)
+- Unified printing/saving method to files/terminal and fixed some differences/bugs - [#1082](https://github.com/qarmin/czkawka/pull/1082)
+- Uses fun_time library to print how much functions take time - [#1082](https://github.com/qarmin/czkawka/pull/1082)
+- Added exporting results into json file format - [#1083](https://github.com/qarmin/czkawka/pull/1083)
+- Added new test/regression suite for CI - [#1083](https://github.com/qarmin/czkawka/pull/1083)
+- Added ability to use relative paths - [#1083](https://github.com/qarmin/czkawka/pull/1083)
+- Fixed stability problem, that could remove invalid file in CLI - [#1083](https://github.com/qarmin/czkawka/pull/1083)
+- Fixed problem with invalid cache loading - [#0000]
 - Fix Windows gui crashes by using gtk 4.6 instead 4.8 or 4.10 - [#992](https://github.com/qarmin/czkawka/pull/992)
 - Fixed printing info about duplicated music files - [#1016](https://github.com/qarmin/czkawka/pull/1016)
 - Fixed printing info about duplicated video files - [#1017](https://github.com/qarmin/czkawka/pull/1017)
diff --git a/README.md b/README.md
index 8e247fe1d..71080b9d2 100644
--- a/README.md
+++ b/README.md
@@ -20,7 +20,7 @@
   - Temporary Files - Finds temporary files
   - Similar Images - Finds images which are not exactly the same (different resolution, watermarks)
   - Similar Videos - Looks for visually similar videos
-  - Same Music - Searches for music with the same artist, album etc.
+  - Same Music - Searches for similar music by tags or by reading content and comparing it
   - Invalid Symbolic Links - Shows symbolic links which point to non-existent files/directories
   - Broken Files - Finds files that are invalid or corrupted
   - Bad Extensions - Lists files whose content not match with their extension
diff --git a/czkawka_core/src/duplicate.rs b/czkawka_core/src/duplicate.rs
index cf14bd7a2..96a34f4fa 100644
--- a/czkawka_core/src/duplicate.rs
+++ b/czkawka_core/src/duplicate.rs
@@ -1,3 +1,4 @@
+use std::collections::HashMap;
 use std::collections::{BTreeMap, HashSet};
 use std::fmt::Debug;
 use std::fs::File;
@@ -424,11 +425,14 @@ impl DuplicateFinder {
             debug!("prehash_load_cache_at_start - started diff between loaded and prechecked files");
             for (size, mut vec_file_entry) in mem::take(&mut self.files_with_identical_size) {
                 if let Some(cached_vec_file_entry) = loaded_hash_map.get(&size) {
-                    // TODO maybe hashset is not needed when using < 4 elements
-                    let cached_path_entries = cached_vec_file_entry.iter().map(|e| &e.path).collect::<HashSet<_>>();
+                    // TODO maybe hashmap is not needed when using < 4 elements
+                    let mut cached_path_entries: HashMap<&Path, FileEntry> = HashMap::new();
+                    for file_entry in cached_vec_file_entry {
+                        cached_path_entries.insert(&file_entry.path, file_entry.clone());
+                    }
                     for file_entry in vec_file_entry {
-                        if cached_path_entries.contains(&file_entry.path) {
-                            records_already_cached.entry(size).or_default().push(file_entry);
+                        if let Some(cached_file_entry) = cached_path_entries.remove(file_entry.path.as_path()) {
+                            records_already_cached.entry(size).or_default().push(cached_file_entry);
                         } else {
                             non_cached_files_to_check.entry(size).or_default().push(file_entry);
                         }
@@ -508,7 +512,7 @@ impl DuplicateFinder {
         debug!("Starting calculating prehash");
         #[allow(clippy::type_complexity)]
         let pre_hash_results: Vec<(u64, BTreeMap<String, Vec<FileEntry>>, Vec<String>)> = non_cached_files_to_check
-            .par_iter()
+            .into_par_iter()
             .map(|(size, vec_file_entry)| {
                 let mut hashmap_with_hash: BTreeMap<String, Vec<FileEntry>> = Default::default();
                 let mut errors: Vec<String> = Vec::new();
@@ -519,15 +523,16 @@ impl DuplicateFinder {
                     check_was_stopped.store(true, Ordering::Relaxed);
                     return None;
                 }
-                for file_entry in vec_file_entry {
-                    match hash_calculation(&mut buffer, file_entry, &check_type, 0) {
+                for mut file_entry in vec_file_entry {
+                    match hash_calculation(&mut buffer, &file_entry, &check_type, 0) {
                         Ok(hash_string) => {
-                            hashmap_with_hash.entry(hash_string.clone()).or_default().push(file_entry.clone());
+                            file_entry.hash = hash_string.clone();
+                            hashmap_with_hash.entry(hash_string.clone()).or_default().push(file_entry);
                         }
                         Err(s) => errors.push(s),
                     }
                 }
-                Some((*size, hashmap_with_hash, errors))
+                Some((size, hashmap_with_hash, errors))
             })
             .while_some()
             .collect();
@@ -581,11 +586,14 @@ impl DuplicateFinder {
             debug!("full_hashing_load_cache_at_start - started diff between loaded and prechecked files");
             for (size, mut vec_file_entry) in pre_checked_map {
                 if let Some(cached_vec_file_entry) = loaded_hash_map.get(&size) {
-                    // TODO maybe hashset is not needed when using < 4 elements
-                    let cached_path_entries = cached_vec_file_entry.iter().map(|e| &e.path).collect::<HashSet<_>>();
+                    // TODO maybe hashmap is not needed when using < 4 elements
+                    let mut cached_path_entries: HashMap<&Path, FileEntry> = HashMap::new();
+                    for file_entry in cached_vec_file_entry {
+                        cached_path_entries.insert(&file_entry.path, file_entry.clone());
+                    }
                     for file_entry in vec_file_entry {
-                        if cached_path_entries.contains(&file_entry.path) {
-                            records_already_cached.entry(size).or_default().push(file_entry);
+                        if let Some(cached_file_entry) = cached_path_entries.remove(file_entry.path.as_path()) {
+                            records_already_cached.entry(size).or_default().push(cached_file_entry);
                         } else {
                             non_cached_files_to_check.entry(size).or_default().push(file_entry);
                         }
diff --git a/czkawka_core/src/similar_images.rs b/czkawka_core/src/similar_images.rs
index c08017243..c797522ca 100644
--- a/czkawka_core/src/similar_images.rs
+++ b/czkawka_core/src/similar_images.rs
@@ -783,7 +783,7 @@ impl SimilarImages {
         // Validating if group contains duplicated results
         let mut result_hashset: HashSet<String> = Default::default();
         let mut found = false;
-        // dbg!(collected_similar_images.len());
+
         for vec_file_entry in collected_similar_images.values() {
             if vec_file_entry.is_empty() {
                 println!("Empty group");
@@ -1338,7 +1338,6 @@ mod tests {
 
             similar_images.find_similar_hashes(None, None);
             let res = similar_images.get_similar_images();
-            // dbg!(&res);
             assert!(res.is_empty());
         }
     }


Test patch: