How to merge many forks from github, 108 many? - Page 1
Forum section: electronics / general

author: RoGeorge
author_rank: Super Contributor
timestamp: November 04, 2022, 05:23:48 pm
content: Official but neglected master projecthttps://github.com/python-ivi/python-iviMain contributor forkhttps://github.com/alexforencich/python-iviAll the 108 forks on githubhttps://github.com/alexforencich/python-ivi/network/membersThe project is a Python module, and so far it seems to be the most complete implementation of the IVI standard for controlling measuring instruments, open and multiplatform. I'm interested mostly in those forks that include Rigol instruments because this is the only brand of SCPI instruments I have so far.Is it reasonable to try to merge all the 108 forks (many don't have the Rigol instruments I need)?Would merging all that be a one hour task, a one month, or rather impossible?What to do to get a reasonably updated version?

author: alm
author_rank: Super Contributor
timestamp: November 04, 2022, 06:20:52 pm
content: I haven't looked at the forks recently, but anything that adds or improves a single instrument driver would be fairly straightforward to merge, while anything making more sweeping changes would be difficult. So all 180 would probably be days of work even for an experienced developer, and possibly more, but if you do a limited subset you might be able to do something in a couple of hours if you've done this before and are comfortable with the git command line.I was at one point considering offering to help maintain and at least merge PRs that are complete with tests etc, since I wrote a number of instrument drivers myself. But I don't have the time right now.

author: RoGeorge
author_rank: Super Contributor
timestamp: November 05, 2022, 07:59:51 am
content: Thanks for the advice. Now I remember we talked about the same abandoned proj, this python-ivi, about two years ago:https://www.eevblog.com/forum/metrology/software-for-automated-testing-(scpi)/msg3398008/#msg3398008and I might still have the tools I used back then to identify the forks that have Rigol drivers.Back then only 10 forks were having drivers for Rigol.Will post in the old thread in case I'll try to merge the relevant forks again.

author: DiTBho
author_rank: Super Contributor
timestamp: November 05, 2022, 04:58:27 pm
content: Which wave gen and dso do you need? (Rigol parts)It should be cleaner to write dedicated drivers.Sigrock should have something.

author: RoGeorge
author_rank: Super Contributor
timestamp: November 05, 2022, 08:14:14 pm
content: I have a DSO from the DS1000Z series, DS1104Z, an AWG from the DG4000 series, DG4202, and a power supply from the DP800 series, DP832A, all 3 from Rigol.The ones I was needing were the IVI drivers (python-IVI drivers more precisely) for the AWG and for the DSO. Such an IVI driver is suppose to translate an IVI command to lower SCPI commands specific to each instrument.I'm not sure Sigrok is using Python-IVI at all, I think it doesn't use IVI commands at all, there are other kind of drivers.In IVI, the word driver is somehow misleading, it's not like a mouse driver which is OS specific. IVI drivers are, briefly said, like a dictionary between a generic set of IVI commands (e.g. all instruments of type "Oscilloscope" will implement the same sets of IVI commands no mater what model the oscilloscope is, or who made it, while for SCPI commands each model uses its own set of SCPI commands that are not compatible between models or between manufacturers) and the specific SCPI corresponding commands of a particular model of instrument.Unfortunately, the IVI standard was defined for Microsoft's C# or .Net, and the python-IVI implementation will never be 100% compatible because of some language specific properties that are different between Python and .Net or C#. That is why, the IVI drivers offered by the instrument's manufacturer are useless for the Python-IVI implementation of the IVI protocol.Might be interesting to try to run the manufacturer's (Microsoft) IVI drivers (in Linux) using WineHQ, or mono, but even if that would work, it looks more like a stunt, will have yet another proprietary binary blob that might not even work at a future time.So far I've temporary parked the forks merging, and spent all my time with the other topic,https://www.eevblog.com/forum/programming/extract-precise-amplitude-and-phase-from-a-frequency-sweep-(vna-from-dsoawg)/

author: Sherlock Holmes
author_rank: Frequent Contributor
timestamp: November 22, 2022, 10:20:46 pm
content: Quote from: RoGeorge on November 04, 2022, 05:23:48 pmJust to clarify, you don't merge forks in Github, you merge branches. A branch might well be in a fork, but all merging is at branch level.

author: Sherlock Holmes
author_rank: Frequent Contributor
timestamp: November 22, 2022, 10:25:07 pm
content: Quote from: RoGeorge on November 04, 2022, 05:23:48 pmThere are a lot of forks and some of them now have multiple branches and some of the forks are forks of other forks.I don't think Github lets us take a birds eye view of all branches and all forks, that would certainly help answer your question.But here's an interesting factoid about Github, all commits, on every single fork, are all actually in the original repo. The forks are "virtual" repos so to speak, they are not distinct repos as is often assumed.This isn't much help with your question though, I'll need to think about this, its a good question.

author: Sherlock Holmes
author_rank: Frequent Contributor
timestamp: November 22, 2022, 10:33:46 pm
content: Frankly this is complex, to all intents and purposes you're looking at a large number of divergent and possibly incompatible variants of the original repo.In addition to which you must factor in the fact that each fork was created at a different date and time and so each fork - could - be a fork of the original repo at a different state...Also, although you have a fork and have the most recent activity of any fork, you do not have this commit (one example among many)https://github.com/acconeer/python-ivi/commit/df7945df377b253955efa0c25e3f2c8f8f7eec4cThis is a serious problem with Github and open source, unless fork ownersandrepo owners stay on the ball, the code just slowly bifurcates and there is no "current" version or "best" version, ideally contributors would be creating PRs back to the main repo and the repo owner would be diligent enough to either merge or provide feedback, fork owners should also regularly (routinely) pull from the main repo too to pickup whatever has been merged from other contributors, but this is often never done.

author: SiliconWizard
author_rank: Super Contributor
timestamp: November 22, 2022, 10:44:50 pm
content: Yeah. That's something I pointed out a while ago regarding how people use github in general and how inefficient it is when projects get split up across tens or hundreds of forks. Nobody will ever care or even have the time to consolidate any of that. It's just not doable for maintainers. For mere "users", you have to pick one of the forks that's closest to what you need and stick to it.Even if you managed to properly merge 108 branches, which could take weeks, while not introducing horrible and intractable bugs, that would be a one-time affair. And now what? Would you keep doing that on regular basis watching modifications across 108 forks (which may each have several active branches?)I have no secret sauce for that. I tend to only use open-source projects that have active official maintainers and stick to the official project. Too much of a mess otherwise. For the rest, I write my own stuff.

author: Sherlock Holmes
author_rank: Frequent Contributor
timestamp: November 22, 2022, 10:49:47 pm
content: Quote from: SiliconWizard on November 22, 2022, 10:44:50 pmIt's kind of a shame, there are probably umpteen very good disparate changes scattered across some of these forks. alas they will remain forever unknown!

author: RoGeorge
author_rank: Super Contributor
timestamp: November 22, 2022, 10:50:11 pm
content: Quote from: Sherlock Holmes on November 22, 2022, 10:25:07 pmIt has something, it's under the 'Insights' tab of each repo:https://github.com/python-ivi/python-ivi/network/membershttps://github.com/python-ivi/python-ivi/network

author: alm
author_rank: Super Contributor
timestamp: November 23, 2022, 08:32:55 am
content: Quote from: Sherlock Holmes on November 22, 2022, 10:25:07 pmAre you saying a Github fork is not like a Git clone, in creating a copy of the original repository that references the original, like what the documentation says? Do you have a reference for how this works on a Git level? This seems like an awful lot of effort for them to go through beyond the basic Git functionality just to safe some duplication. It would also imply all forks disappear when the original repository is deleted.If I were them I'd just implement some object-level deduplication agnostic of any repository boundaries.

author: ledtester
author_rank: Super Contributor
timestamp: November 23, 2022, 11:45:17 am
content: Quote from: Sherlock Holmes on November 22, 2022, 10:25:07 pmHere is an example of this...This fork:https://github.com/jkc-sw/python-iviis 4 commits ahead of the original repo:https://github.com/python-ivi/python-iviand this is the latest commit on the jkc-sw repo:https://github.com/jkc-sw/python-ivi/commit/5f8356ebd39041673b5546d77bc202295550fbe1Now replace the "jkc-sw" in the url with "python-ivi":https://github.com/python-ivi/python-ivi/commit/5f8356ebd39041673b5546d77bc202295550fbe1and github will show you the same commit but also report:QuoteThis suggests that the jkc-sw and python-ivi repos are sharing the same physical git repo.Quote from: alm on November 23, 2022, 08:32:55 amYou would only delete the underlying git repo when all forks have been deleted - like a reference counting / garbage collection type of situation.

author: Sherlock Holmes
author_rank: Frequent Contributor
timestamp: November 23, 2022, 05:46:07 pm
content: Quote from: alm on November 23, 2022, 08:32:55 amYes Sir, aGitHubfork (as you know, forks are not part if Git itself) is essentially a set of user specific branches in the main repo, there is only one repository, the repo is not in any sense copied when we fork it.Have you noticed how fast it is to create a fork? even of a huge repo?The evidence for this is found when one ues Octokit to access and pull repository metadata, if you do that you'll be able to see that commits that exist in a fork also exist in the main repository, they are present in that main repository.I think that under the hood forks are just per-user branches where the user name is invisible. So "Janet" forking some repo that has a "master" and "version2" branch actually is just the creation of a "Janet.master" and "Janet.version2" branches and the system/UI etc simply hides the "Janet." part of the name.In this sense forks are "virtual repos" nothing but a set of branches, physically copying repos strikes me as a terrible idea and one that could lead to hugely complex confusion if things ever went wrong, the "virtualizing" is absolutely the way to go.

author: alm
author_rank: Super Contributor
timestamp: November 24, 2022, 01:05:48 am
content: Quote from: ledtester on November 23, 2022, 11:45:17 amInteresting. This is little help when merging from the command line, however, as you would be doing when merging a hundred or so different repositories, because the commit is not included when cloning the python-ivi repository:Code:% git clone https://github.com/python-ivi/python-ivi.gitCloning into 'python-ivi'...remote: Enumerating objects: 4901, done.remote: Total 4901 (delta 0), reused 0 (delta 0), pack-reused 4901Receiving objects: 100% (4901/4901), 1.25 MiB | 3.84 MiB/s, done.Resolving deltas: 100% (3985/3985), done.% cd python-ivi% git show 5f8356ebd39041673b5546d77bc202295550fbe1fatal: bad object 5f8356ebd39041673b5546d77bc202295550fbe1Only when cloning the jkc-sw repository will the 5f835 commit be included. So as far as Git is concerned they are two separate repositories with two different sets of commits.

author: ledtester
author_rank: Super Contributor
timestamp: November 24, 2022, 01:50:42 am
content: I think git fetch will do the job for you. See this SO discussion:https://stackoverflow.com/a/2949855/866915

author: RoGeorge
author_rank: Super Contributor
timestamp: November 24, 2022, 06:58:18 am
content: For this particular merge I was interested mostly in those forks containing Rigol drivers, which drivers happen to be in a specific folder named rigol.I've copy/pasted all the user names from the forks webpage, concatenated the list with mkdir and git clone, then cloned all the 108 forks locally. Then made a search on the local disk, and eliminate those forks without the rigol directory. Only 10 forks remained, which I diff with 'meld', looking only at the Rigol instruments files. All 10 were about the same so very easy to merge, the most recent/active fork for the Rigol drivers was the one from 'nicedwarf'.Then I've made a 109'th fork, then realized it's not possible any longer to submit changes from a local git back into github, because 'security' and extra hoops with security tag from Microsoft and so on, so I've said good-bye github, and uploaded my fork with its few changes to gitlab instead of github:https://gitlab.com/RoGeorge/python-iviGitlab is still sane, thank you.But then I've realized it would be easier to not use any of the python-ivi, pyvisa, pyvisa-py, python-vxi11, instrument specific drivers and/or proprietary VISA shared libs at all, and send SCPI commands to SOCKET instead.

author: DiTBho
author_rank: Super Contributor
timestamp: November 24, 2022, 10:23:41 am
content: Quote from: RoGeorge on November 24, 2022, 06:58:18 amso, is it the definitive approach?you made things very confused!

author: RoGeorge
author_rank: Super Contributor
timestamp: November 24, 2022, 12:55:30 pm
content: Other protocols have their own pros and cons, depending on the end user, and depending on the end application.- SCPI is plain text, trivially simple to use, but implies taking care of details, like dealing with possible errors, or learning the exact SCPI commands known by the instrument, or setting the instrument address and so on.- VISA will abstract the physical layer, and present the same software interface no matter how you want to talk with your instrument, either through LAN, USB, serial port or GPIB. VISA has some extra features regarding the messages exchange, too, not only about abstracting the physical layer. All that might lower the level of details needed to be known by the end user. I only use LAN connectivity, with fixed IP addresses, don't need to discover my instruments in the LAN, and already know their capabilities.- IVI will abstract the instrument model/manufacturer, and try to make all the oscilloscopes the same, all the generators the same, and so on. IVI will also unify the various SCPI dialects into a unique set of IVI commands. For that, it will need some sort of translator from IVI to SCPI instrument specific, so it will need an "IVI driver" (better name would have been "IVI translator"), which IVI driver is specific to a given model of instrument. So apart from the hardware drivers (if needed), now an IVI driver is needed, too. This IVI driver is specific to the IVI implementation, same as a hardware driver is specific to each OS. You will need IVI drivers written specifically for your instrument AND for python-ivi. For example, the COM-IVI drivers provided by the instrument's manufacturer are useless for python-ivi.IVI and VISA might make sense in an industrial setup, where instruments can be configured by different operators, can be moved around from one lab to another, or can be replaced with newer models, and so on. In a personal lab, where the instrument never changes, there is not much advantage of adding the extra layers.I've tried IVI, and almost any setting is obfuscated by 2-3 layers of abstraction. Had to dig through code sources way too often to learn how to change otherwise trivial settings. Then, found myself sending IVI commands while sniffing my own TCP packets to know what SCPI commands were sent by IVI, and if they correspond or not with what I was trying to do.IVI and VISA are OK as industry standards, just not good for my personal use case.Maybe I should write a step by step 101. LXI-ready instruments are trivial to control remotely. It all works out of the box with nothing but a LAN cable, no other software or drivers are needed. Telnet to the instrument and type SCPI commands, or open a TCP socket from a programming language, and send SCPI from there.

author: DiTBho
author_rank: Super Contributor
timestamp: November 24, 2022, 06:23:36 pm
content: I think one should focus only on the needed operations.- frequency/div setup- amplitude/div setup- sample record- sample dump

author: SiliconWizard
author_rank: Super Contributor
timestamp: November 24, 2022, 07:00:46 pm
content: To sum up: write your own code and you'll have wasted a lot less time.

author: alm
author_rank: Super Contributor
timestamp: November 24, 2022, 09:48:41 pm
content: Quote from: SiliconWizard on November 24, 2022, 07:00:46 pmThat's always what it looks like at the beginning. Surely writing it from scratch will be faster and lead to a better result. Sometimes it's true, but it's equally common to after some weeks of work discover that reinventing the wheel is a lot more complicated than you thought, and the system you built might have ended up equally complicated as the complicated thing you tried to avoid.

author: SiliconWizard
author_rank: Super Contributor
timestamp: November 24, 2022, 10:38:04 pm
content: Quote from: alm on November 24, 2022, 09:48:41 pmWell, I don't really agree with that. I would say almost the opposite: that's almost always what it looks like at the end.At the beginning, code that appears to have been used by other people looks like integrating it to get what you want should be a breeze.Quote from: alm on November 24, 2022, 09:48:41 pmI'm not advocating reinventing the wheel when it doesn't make sense of course, but IME developers tend to shy away from writing their own code even for relatively basic functionality because they have been told it was bad not to reuse others' work. (And, sometimes - what I'm gonna say is politically incorrect - that's because they just are not capable of doing so.) So there you go, with all the bloat that follows and hours of debugging and managing versions of third-party stuff you have no control over. Especially when it's "random" online projects and all the worse if they are spread across hundreds of forks.Now admittedly, and maybe that's what you had in mind as a counter-example, OTOH software developers have a tendency to want to write their own code for everything and discard others' code. So that's a constant battle between the two opposites.

author: ledtester
author_rank: Super Contributor
timestamp: November 24, 2022, 11:48:41 pm
content: Here's an example of using git fetch to perform a merge of two repos. The commit "5f8356" isn't in python-ivi's tree but is in jkc-sw's tree so the "git show" command before the fetch fails but succeeds afterwards.Code:PS C:Temp> git clone https://github.com/python-ivi/python-ivi.gitCloning into 'python-ivi'...remote: Enumerating objects: 4901, done.remote: Total 4901 (delta 0), reused 0 (delta 0), pack-reused 4901Receiving objects: 100% (4901/4901), 1.21 MiB | 1.91 MiB/sReceiving objects: 100% (4901/4901), 1.25 MiB | 1.65 MiB/s, done.Resolving deltas: 100% (3985/3985), done.PS C:Temp> cd .\python-ivi\PS C:Temp\python-ivi> git show 5f8356fatal: ambiguous argument '5f8356': unknown revision or path not in the working tree.Use '--' to separate paths from revisions, like this:'git <command> [<revision>...] -- [<file>...]'PS C:Temp\python-ivi> git fetch https://github.com/jkc-sw/python-ivi.gitremote: Enumerating objects: 20, done.remote: Counting objects: 100% (12/12), done.remote: Total 20 (delta 12), reused 12 (delta 12), pack-reused 8Unpacking objects: 100% (20/20), 4.52 KiB | 53.00 KiB/s, done.From https://github.com/jkc-sw/python-ivi* branch HEAD -> FETCH_HEADPS C:Temp\python-ivi> git show 5f8356commit 5f8356ebd39041673b5546d77bc202295550fbe1Author: jkctaiwan <guanru0919@yahoo.com>Date: Fri Mar 29 14:39:16 2019 -0700explicit on verb and headerdiff --git a/ivi/tektronix/tektronixMDO4104C.py b/ivi/tektronix/tektronixMDO4104C.pyindex 3ba6fbd..dcec4e3 100644--- a/ivi/tektronix/tektronixMDO4104C.py+++ b/ivi/tektronix/tektronixMDO4104C.py@@ -70,10 +70,10 @@ class tektronixMDO4104C(tektronixMDO4000, tektronixMDOAFG):self._write(":data:width 2")self._write(":data:start 1")self._write(":data:stop 1e10")- # eanble verbosity+ self._write(":VERBose ON")^Mself._write(":HEADer ON")...rest of commit...PS C:Temp\python-ivi>

author: alm
author_rank: Super Contributor
timestamp: November 25, 2022, 10:42:22 am
content: Sure, you could also do that with separate copies (clones) of the repositories. But what's the point? Why not add the other repo as a remote and merge/rebase on that?

author: tszaboo
author_rank: Super Contributor
timestamp: November 25, 2022, 12:13:39 pm
content: You know what they say: How to eat an elephant?

author: DiTBho
author_rank: Super Contributor
timestamp: November 25, 2022, 12:19:06 pm
content: @RoGeorgeSpent my Black Friday bonus and Bought a 1104z dso, I can help.I d like to implement stuff in C.

author: RoGeorge
author_rank: Super Contributor
timestamp: November 25, 2022, 01:44:44 pm
content: Congratz for the new oscilloscope!It's about 10-30 lines of Python, but sure can be done in C.As an inspiration you can take a look athttps://github.com/lxi-tools/liblxi.But before that, assign a fixed IP to your new dso, then open a telnet to the IP you assigned, port 5555. Mine has 192.168.1.3, so with netcat that will be:Code:nc 192.168.1.3 5555aaa@bbb:~$ nc 192.168.1.3 5555# then from inside nc, type*IDN?# and if all OK, your oscilloscope will answer with something likeRIGOL TECHNOLOGIES,DS1104Z,DS1ZA123456789,00.04.05.SP2That's it, you now officially know how to talk SCPI with your oscilloscope, in this case using Telnet.

author: Sherlock Holmes
author_rank: Frequent Contributor
timestamp: December 05, 2022, 03:58:55 pm
content: I think if one forks a repo that isn't clearly and regularly maintained then you're on you're own. You've basically taken their code and taken responsibility for it's future.It's incredible how many repos are out there with issues dating back months that the owners haven't even responded to.