[solved] how do you compare two folders? to extract source diff - Page 1
Forum section: electronics / general

author: DiTBho
author_rank: Super Contributor
timestamp: July 11, 2021, 03:33:43 pm
content: Usually when I work with Linux kernels, I download the source, and immediately create a git branch labelled "original code", so I can commit my changes on it, and this way it's easy to extract patches.Now I am working on a two and large kernels, my colleagues didn't use Git, and differences between kernel-A and kernel-B are unclear.Kernel-A and kernel-B are derived from the same source, but they are two different branches, both bugged, and with different working and not working stuff.Within each main folder, the sources are distributed in several sub-folders, with a deep hierarchy. I see that some sub-folders have been renamed, deleted or added, as well as some of the contained files.This is bad without Git, because it's not immediately clear how to know how a file was renamed or where it was moved.How would you compare two folders, to extract diff, in this case?

author: esepecesito
author_rank: Regular Contributor
timestamp: July 11, 2021, 04:09:28 pm
content: If they are not completely different "ls -lR > /tmp/old" the same for the other version, in another file, then diff. Crude, but may work.Another possibility with for:for file in d1/*.cpp; dodiff "$file" "d2/${file##*/}"doneYou can also do it with find, connot do it out of my head now... but something like:find /path/to/src -exec diff {} d2/${file##*/} \;I'm sure it's wrong... but something along the lines should do.

author: SiliconWizard
author_rank: Super Contributor
timestamp: July 11, 2021, 05:32:00 pm
content: Any decent diff tool will work on directories and recursively.But, well, I don't know how different your two kernels are. But good luck analyzing every single difference in all source files. That could be a very time-consuming work.

author: magic
author_rank: Super Contributor
timestamp: July 11, 2021, 06:06:50 pm
content: Kids these daysQuoteeditQuote from: DiTBho on July 11, 2021, 03:33:43 pmNevermind, that's gonna suck big time. Not sure what could find renamed files and directiories, particularly if they have been modified too.

author: SiliconWizard
author_rank: Super Contributor
timestamp: July 11, 2021, 06:25:35 pm
content: Ah yes, if there are renamed files and directories, all the worse.A decent VCS can handle renamed stuff and keep track of what has been renamed to what.Now working on such a large code base as the Linux kernel without any VCS is complete madness.

author: DiTBho
author_rank: Super Contributor
timestamp: July 11, 2021, 06:30:36 pm
content: Quote from: evb149 on July 11, 2021, 05:07:56 pmNo, they do come neither from git nor from svn, that's exactly the problem.There is no history, just files and folders, and I have to somehow reconstruct the history.There is also a big problem: I should need something like "bisect" (which is a common tool with Git, at least for Linux), because there are things that work with kernel-A but not with kernel-B, and vice-versa.Frankly, it's a full mess ... it seems like a punishment, I must have done something bad in my life to deserve it

author: SiliconWizard
author_rank: Super Contributor
timestamp: July 11, 2021, 06:43:03 pm
content: I would first use a tool or write a script to compare directories and file names, and give you what differs. At this point, you can just compare names, and list files and directories that exist on one side and not the other. Some diff tools have this feature. That will give you new and renamed files/directories. If there aren't too many of them, reconstructing what is just a rename and what is new should not take too much time.Next step would be to do a full diff on all files. You can rename the renamed files/directories back to their original name before that, or you can just leave them as is and diff them separately.

author: Doctorandus_P
author_rank: Super Contributor
timestamp: July 11, 2021, 09:13:57 pm
content: I like:http://meldmerge.org/It's not the greatest source code comparison program, and it's written in python, so it is quite slow, but it is open source and does work.It can also compare sub directories recursively, and it's easy to click around with a mouse.It does not work well with renamed directories though. It will find the different directory names, but then does not compare the files in those directories.

author: DiTBho
author_rank: Super Contributor
timestamp: July 11, 2021, 09:39:28 pm
content: Thanks guys, I feel really lost.I just bought a copy with license of "Beyond Compare Pro" for Windows. Its activation key will be dispatched by email probably tomorrow because it requires human interaction. I will use it over Samba on a laptop, with Meld and MeldMege locally on the Linux server.I read good reviews about, it's a bit pricey but it seems to have a lot of serious and advance features.We will see.

author: AntiProtonBoy
author_rank: Frequent Contributor
timestamp: July 12, 2021, 01:20:50 am
content: I think Double Commander should be able to visually compare files for you.Example:https://doublecmd.sourceforge.io/static_gallery_mirror/page278_1.html

author: DiTBho
author_rank: Super Contributor
timestamp: July 12, 2021, 09:02:16 am
content: Quote from: AntiProtonBoy on July 12, 2021, 01:20:50 amoh, nice! and it's written in fpc (freepascal)

author: SiliconWizard
author_rank: Super Contributor
timestamp: July 12, 2021, 06:06:03 pm
content: Now that you invested in Beyond Compare... not much point in using other, less powerful compare tools. I for one have been using WinMerge (on Windows) for years. Works fairly well, and still maintained. Not as many features as Beyond Compare though.https://github.com/WinMerge/winmerge/releases/As I said, I would personally have still probably written tools/scripts to "preprocess" the huge amount of directories and files, and after that, I would probably have used a visual compare tool such as the above for actually merging code in source files that are different. I think that would have saved time in the end. But I sort of like writing tools, so...

author: 2N3055
author_rank: Super Contributor
timestamp: July 12, 2021, 06:34:23 pm
content: Quote from: SiliconWizard on July 12, 2021, 06:06:03 pm1+ for WinMerge.

author: DiTBho
author_rank: Super Contributor
timestamp: July 12, 2021, 07:07:23 pm
content: Quote from: SiliconWizard on July 12, 2021, 06:06:03 pmWith which algorithms? and to do what?Yesterday, while I was waiting for my license to be activated, I wrote a C program that can find "added" and "removed" files and folders, but the real problem is understanding "what" a file or folder was renamed to.My program can index each file, and I can re-use some old A.I. algorithms to extract a "match percentage", say document X in kernel-A is probably the ancestor of document Y in Kernel-B, documents have different file-names, the match percentage is ..."Beyond Compare" does this analysis. I don't know how it does it

author: AntiProtonBoy
author_rank: Frequent Contributor
timestamp: July 13, 2021, 02:35:11 am
content: Total commander is also a good one for windows users.

author: SiliconWizard
author_rank: Super Contributor
timestamp: July 13, 2021, 03:43:28 pm
content: Quote from: DiTBho on July 12, 2021, 07:07:23 pmI think I said it already? First comparing files and folders, and just output the ones that are different on each side. Without trying to automatically figure out which are potential renames, and which are just completely different. Of course there are other tools that can do this. Winmerge can also do that for you. But writing your own just gives you the opportunity to list differences in the exact way that you want, and that will make your life easier. That's the whole point. Most visual diff tools will just give you a long list of differences, which, when there are many files and folders, is not necessarily the best way to deal with that.Quote from: DiTBho on July 12, 2021, 07:07:23 pmNow if you meant - which is I guess why you asked which algorithms I would use - not just listing which files or directories are different, but also determining which are potential renames, then that's more complex of course. If Beyond Compare does this, then it can be handy. But, I was just assuming that (unless the teams working on code were completely nuts) the number of file/directory renames was likely small compared to the total number of files and directories, and thus, that presenting differences in a way most convenient for you was more important than trying to automatically determine which were renames and which were just new stuff, something that I suggested doing manually in the end.Now if Beyond Compare does everything you want and presents things in a convenient way for you, that's all good.

author: DiTBho
author_rank: Super Contributor
timestamp: July 13, 2021, 07:02:00 pm
content: I'm pretty tired, I've been working on this thing non-stop for two days, let me say ... Beyond Compare is really worth all the money it costs, it's really super helpful!It allowed me to somehow bisected and isolated 84 files which are related and differ in changes for the most; then cross-comparing things (between kernel A and Kernel B), I found 5 of them manifest several problems, which are really sillya PHY not correctly initialized, the eth0 was up and seemed to work, but unable to communicate with the physical layera "UNIX-V" flag not correctly checked, and misspelled, this somehow passed the compile-check but then the kernel crashed as soon as it tried to mount somethinga script trying to redefine CC but in the wrong way, this dropped all the passed cflags and made the compiler to compile a couple of modules in the wrong waya couple of very bad bad mistakes in the DTB file, two fields appeared truncated probably due to a bad copy&pasteReally silly mistakes, I think was lucky to catch them so fast. Now I need to sweep out the garbage and all the temporary debug flags from the final trunk; I am in the process of creating a git-repo for everything, forcing my colleagues to use it.edit:thanks again guys

author: bd139
author_rank: Super Contributor
timestamp: July 13, 2021, 09:19:37 pm
content: Been there. We had a file share and the only person allowed to edit stuff on it had to obtain the wooden spoon. This was until I bought a second wooden spoon and cocked their entire process up. This resulted in us moving to SVN and proper traceable version controlBeyond compare is excellent for mirroring directories to removable media aka backups too. Excellent tool and well worth paying for. Have been a paying customer for about 15 years now