Starting with Kicad...    local / GIT question  - Page 1
Forum section: electronics / general

author: Back2Volts
author_rank: Unknown
timestamp: June 02, 2016, 12:15:43 pm
content: Recently, I have started to play with Kicad. It is my first use of a tool of this kind to draw schematics, etc... I started with the schema and library editors, created some new symbols and placed them in the schema. Minutes ago, I tried to go to the footprint editor for the first time and received a library table warming. Digging into settings I have noticed a GIT URL in what looks like a path.I am GIT ignorant and learning/using it is not one of my immediate priorities. So my question isDoes Kicad (4.0.2 in Windows7) by default use local directories, GIT or both? Is there a setting to go full local ? Should I be concerned?

author: stmdude
author_rank: Frequent Contributor
timestamp: June 02, 2016, 02:38:42 pm
content: The footprint libraries are not part of the installation.You can use git (as you've seen) to "download" a copy of the latest-and-greatest that the KiCad guys have, and to update when you think there's something new available.Once you've downloaded the footprint libraries, you're working with local files.

author: Back2Volts
author_rank: Unknown
timestamp: June 02, 2016, 03:56:41 pm
content: Thank you!Are there any libraries for toggle switches ? I only see push-buttons.

author: stmdude
author_rank: Frequent Contributor
timestamp: June 03, 2016, 06:02:59 am
content: No idea. I've resorted to drawing most of my own footprints..

author: timofonic
author_rank: Frequent Contributor
timestamp: June 03, 2016, 02:59:13 pm
content: It's is possible to use Git for schematic and board designs inside KiCad directly?This would provide infinite undo, tagging new changes and a collaborative environment. This can be omething very good for true OSHW, please don't relate it to shady organizations like OSHWA but CERN.Quote from: stmdude on June 03, 2016, 06:02:59 amWhat about sharing them in GitHub too? KiCad-library Team could cherry pick them too

author: stmdude
author_rank: Frequent Contributor
timestamp: June 04, 2016, 08:25:07 am
content: Quote from: timofonic on June 03, 2016, 02:59:13 pmYou certainly can use git as revision-control for KiCad schematics and boards, but it's not integrated into KiCad. You'll need to use something like TortoiseGIT (or command-line git) to do it.Quote from: timofonic on June 03, 2016, 02:59:13 pmMaybe once the new component library system gets in place. Right now, adding ~100 very specific chips would only create clutter in the library. They're working on the new component library, so it shouldn't be long.

author: timofonic
author_rank: Frequent Contributor
timestamp: June 04, 2016, 03:01:08 pm
content: Quote from: stmdude on June 04, 2016, 08:25:07 amNot tightly integrated = uselessAt least in my opinion.What new library system? Are they going to use one work advanced taxonomy and database like high-end EDA software.

author: stmdude
author_rank: Frequent Contributor
timestamp: June 05, 2016, 04:08:27 pm
content: Quote from: timofonic on June 04, 2016, 03:01:08 pmThey're going to use the same model as they use for footprints, but for schematic symbols as well. More of a table-view than tree-view.

author: bson
author_rank: Unknown
timestamp: June 05, 2016, 11:13:29 pm
content: Quote from: stmdude on June 04, 2016, 08:25:07 amYou CAN but don't expect it to work all that great between multiple computers. The reason is KiCAD stores absolute paths (as reported by realpath()) in most locations. Unless you make sure to exactly match library and local repo locations they won't match, and KiCAD will fall back on the Rescue library. When you then check this in it will have switched from configured libraries to Rescues, and when you check out it will be using the Rescues everywhere. If it can't find the absolute path of the Rescues it will give you a popup and the component or footprint won't be available.Don't ask me how I know this.The fix requires manually editing and making paths relative before checking in, before each change, bypassing all the braindead GUI project management.If this sounds scary, don't use any kind of source control, especially not mixing multiple platforms. On the other hand, figuring out the file formats is easier than figuring out (and remembering) where to find things in the GUI.You can also standardize on say a /Projects directory, but then you invariably run into some tool that won't like that.

author: HackedFridgeMagnet
author_rank: Super Contributor
timestamp: June 05, 2016, 11:28:39 pm
content: Quote from: timofonic on June 04, 2016, 03:01:08 pmNot sure why this is the case?I prefer no integration with the application and tight integration with the file system.It certainly does work and is extremely useful.I've been using SVN or Tortoise SVN for years with Kicad.I could have used GIT and maybe I will change to Git to Get better access to the libraries.

author: timofonic
author_rank: Frequent Contributor
timestamp: June 06, 2016, 03:15:35 am
content: Quote from: HackedFridgeMagnet on June 05, 2016, 11:28:39 pmAre you a software developer? This is how develops think, but I doubt it's useful in a relatively big electronics design group workforce.Why do other high end EDA software do it in totally different ways than you say? Some of them even provide very interesting RTCE features

author: HackedFridgeMagnet
author_rank: Super Contributor
timestamp: June 06, 2016, 03:49:52 am
content: I will rephrase my question. As you seemed to be answering something else?In the context of version control software working with ED software, why do you say "not tightly integrated = useless"?To help you with your answer, it isn't useless, because I and other people find it useful.It's that easy. Fairly simple logic from someone who (incidentally) has done a fair bit of software development.Quote from: timofonic on June 06, 2016, 03:15:35 amIDK the answer to this but it doesn't make my approach useless.Personally I am not a big fan of highly intertwined (tangled) applications but I can see that in some instances that it works well.Here's a smiley.

author: stmdude
author_rank: Frequent Contributor
timestamp: June 06, 2016, 07:10:29 am
content: Quote from: bson on June 05, 2016, 11:13:29 pmYup, I've had that problem. Solved it twice, actually.First solution was to have all my KiCad libraries/footprints/project on a file-share at home, and map it as the same drive on all machines. For laptops, I turned on "Offline files" for the share, so I could work away from my home. That way, all computers have the exact same path for everything KiCad related.Second solution was needed when I introduced Linux into the mix. Enviroment variables to the rescue. Basically, I used a "MY_KICAD" variable to point to the same root as I used before. That way, Linux doesn't see "Z:\Foo\Kicad", and Windows doesn't see "/mnt/server-homedir/foo" .

author: amspire
author_rank: Super Contributor
timestamp: June 06, 2016, 01:06:34 pm
content: Quote from: timofonic on June 06, 2016, 03:15:35 amI have used SVN in an Animation Studio where there are no programmers/engineers/computer experts amongst the animators. They were basically artists who migrated to PC's from 10 to 30 years of hand drawn animation. Once they were shown what SVN can do, they absolutely loved it. Not one complained. No-one said it was too hard or cumbersome. Every now and then, they would mess things up and I would have to sort it out, but most of the time, it worked brilliantly.A key part is that in one commit, the SVN stores the output of different non-integrated programs together in a single version. If a version system cannot do this, it is not much use in a real production environment. Integrated version control systems are very limited, and it can be very difficult accessing the files when you no longer have a working license to the program with the integrated version control (common now in the era of subscriptions and activations).Couldn't have succeeded without TortoiseSVN to make SVN usable, but I could imagine a better SVN management layer that can prevent you from things like starting an edit without doing an update first.SVN for this use is a far better solution the Git - I wouldn't even consider using Git.Richard.

author: timofonic
author_rank: Frequent Contributor
timestamp: June 07, 2016, 12:28:59 am
content: Quote from: amspire on June 06, 2016, 01:06:34 pmAltium Designer uses SVN. Am I right?What's so wrong about Git?You can't autosave on a VCS, right?

author: amspire
author_rank: Super Contributor
timestamp: June 07, 2016, 12:52:12 am
content: Quote from: timofonic on June 07, 2016, 12:28:59 amGit works for programmers writing text source core files. For general version control work, merge becomes almost useless, and Git is designed around merging. For Animation file use and many other general version control uses, you have to have a single central repository and none of the files can be merged. Merging is never done at all. The Git distributed repository model does not work in practice for Animation files. Also the concepts behind Git are too hard to explain to non-programmers. Explaining SVN is very easy and very intuitive (when kept simple). Essentially, just about everything that Git claims as an advantage over SVN is a disadvantage in uses such as animation files.One of the most useful features of SVN is you can just checkout a sub-directory. So in animation, an animator can check out just one scene of an episode. You cannot do this in Git. You can do a sparse checkout in Git but we are back to concepts that are difficult to explain. The SVN sub-directory checkout is exactly what the animators want.

author: timofonic
author_rank: Frequent Contributor
timestamp: June 07, 2016, 03:19:03 am
content: Quote from: amspire on June 07, 2016, 12:52:12 amCan't these Git procedures be abstracted in some way? Git has the advantage of many free services for it.Why can't merging be useful in EDA? Maybe someone does a different version of the project. Maybe the merging process would need to be very interactive and save each modification at extremely detailed bevel like a database? Just a question and naive idea.

author: amspire
author_rank: Super Contributor
timestamp: June 07, 2016, 03:59:45 am
content: Quote from: timofonic on June 07, 2016, 03:19:03 amThe argument would make sense in cases where free services are any use, but I have never seen it needed in Animation Studios - just want a version control system that works with local servers. That is it.QuoteAnimation programs often can store files in a text format, but they are not designed for manual editing. You do not ever want to get a scene "infected" as a result of a well meaning but invalid attempt to directly edit 2D/3D files. I have used file comparisons to see what is different in two files, but subsequent changes are always done within the editing software packages.Something like photoshop PSD files are much simpler then 2D/3D scene files - how would Git go at merging Photoshop files? How would Git go at merging Kicad schematic and PCB files? Does the concept even make sense? If you had Leonardo Da Vinci and Michaelangelo both working on a version of a painting, and you merged them with Git, would you get a useful result? Merging is an idea only used by programmers - it makes sense in programs.Even in 2D, we often had scenes that were over 1GByte in size with 20,000 files. With SVN, we were able to have an animator commit a scene from home on a slow ADSL connection, and within a few minutes, an animation director was able to open the same scene (with the 20,000 files), make changes, commit, and minutes later again the animator was able to open the directors changes. SVN can be pretty amazing.

author: timofonic
author_rank: Frequent Contributor
timestamp: June 07, 2016, 05:15:41 am
content: What about graphical diff and detecting differences after a common state?A painting isn't the same as a schematic, maybe a PCB is a a bit more similar. But both are vector.But I can't believe that is impossible to solve the problem in a smart way.What about this?Git as a Document Format - Wil Shipley

author: Bassman59
author_rank: Super Contributor
timestamp: June 08, 2016, 11:01:40 pm
content: Quote from: timofonic on June 07, 2016, 12:28:59 amYes.One problem with Altium, at least before they introduced their "Vaults" (which has its own issues) is that it doesn't like the libraries being on a shared drive. We use a workflow with exactly one company-wide integrated library, which is maintained by the main PCB guy. He keeps various symbol and footprint libraries, and when we need a new part, he creates or finds the symbol and the footprint, adds some company-specific details to the symbol, and then recompiles the libraries into the one integrated library, which is put on a network share.The problem is that when more than one person is using the library, he can't make changes to it (it borks things badly). So everyone just grabs the latest version of the library and copies it to their local machine. (He sends around an e-mail when changes are made to the library.) It's a manual process, but it's really only a problem when a footprint error is fixed but someone uses the old version of the library before the fix.So the Altium SVN library takes that "copy to local machine" concept and makes it somewhat better. The library on each user's computer is now a long-term working copy that is checked out of the repository. The librarian makes the change, compiles the library, commits it to the repo, and then the users simply do an $ svn update to grab the latest changes. (And it may even be automated, when you first start Altium it can do that update automatically.)The vault concept takes this repo thing to a new level, where basically changes to the library are pushed out and made available immediately.Also, you can keep binary files (of any sort, not just a PCB design or library) in a Subversion repository. Two things to remember.One is that with text files, Subversion saves differences between revisions when you commit. This saves disk space and makes for speedier network access. If the files are binary, diffs are (for the most part) impossible so each commit sends and saves the entire file.Two, as has been noted, you can't merge binary files. Why is this important? Say you're working on some code with a friend. You both have working copies checked out on your computers. He makes a change and commits it to the repo. You do an $ svn update to get his changes. What happens is that Subversion silently merges his changes into your working copy. Now what happens if you've made changes to your working copy of the file, but you haven't updated to get his latest changes? When you go to commit, you'll get a conflict warning, and you will have to merge the changes manually. (TortoiseDiff and other tools make this relatively easy.) Fix the conflicts, then commit, and good. With binaries, you don't have the ability to diff, so you can't do a manual merge in case of conflicts. The solution is to obtain a "lock" on the file. Only the working copy with the lock can commit changes back to the repo. A repo can be configured so that certain files automatically have a "needs-lock" property on them, which forces the issue. (TortoiseSVN goes a step further: if you check out a file which requires a lock to commit back, that files is marked as read-only, so you can't even save changes to it.)In practice, this is less onerous than it seems, and it forces you to talk to your co-developers so everyone knows what is being worked on.QuoteDepends on who you ask. Git and Subversion have a different philosophy. The main difference is that with Subversion, there is one repository and users check out what they need from that repository, make the changes, then commit them back. The checkout and commit operations require network access to the repository.With git, each user makes a clone of the repository. The user can make changes and commit them to the local clone, and at some point he can request that the local changes be committed back to the main repository.So note that for Subversion, if a user wants to do some experiments and keep track of changes to them, he has to make a branch of the trunk and work on (commit changes to) the branch. When the results are good, the branch needs to be merged back to the trunk. With git, the experiments remain in the user's cloned copy of the repo, so the only time the user interacts with the main repo is when the changes are done. In practice, the result is the same in either case, and it's incumbent on the project maintainers to ensure that commits to the trunk are good.QuoteNo, that's not the point of the VCS. You don't want to commit unless you're sure that your work doesn't break someone else's work. You don't even need autosave -- just program your fingers to do Ctrl+S every change. (I am serious.)That said, some users set up cron jobs (or the equivalent) which will commit changes back to the repo on a regular basis.

author: amspire
author_rank: Super Contributor
timestamp: June 09, 2016, 01:26:42 am
content: Bassman59 - really great explanation. Not much I can addJust thought I would mention the effects of moving to a version management system in an organisation.Before we went to one, people were regularly making many copies of working directories and making many zip copies. If someone was working an a particular project for years, it was not uncommon to see hundreds of folders and zips for the same project spread across many workstations, servers and backups and it was hard to track what the differences were. It was often hard for an outsider to know which folder was the actual production files and which were folders for experimenting, bug fixing and debugging and so on.With SVN, we were able to wipe out all those folders, and at the completion of each project, we knew we could safely wipe out everything because everything needed for the project was stored in the SVN.The reduction is storage space with SVN or Git instead of file storage is phenominal. I have a folder that is storing SVN repositories for animation files that is probably about 200GBytes big. If I checked out all the files it would probably be 10TBytes of disk space (lots of xml-type text files, and lots of very small files that disks systems do not handle well). If I checked out every version of every file, it is probably over 100TBytes.The SVN repository files are also perfect for synching to backup drives. Each commit stores its changes in a new file, and so the sync only has to copy these differential files for each of the day's commits. I have often been able to sync the 200TByte SVN repository folder for the day in a minute or two.We are in Australia, and at one time when we were working with a Canadian studio, I was able to set up a remote server in the US, sync our Australian based SVN repositories to the US one, and then from the US server, extract the files and send it to the Canadian studio at a many times the speed then I could possibly do from Australia. Even though we only had an ADSL connection in Australia with a 100KByte/sec upload speed, I was able to synch a 20G compressed project to the US server and ship the files to the customer in less then an hour.Third thing that really puts the icing on the cake is that you can use web-dav to make the whole SVN browseable with a web browser, and if you only need an individual file, you can download it with the browser. If you need a whole folder, copy the browser address, open TortoiseSVN, and click export or checkout. The copied address is automatically pasted into the SVN address field, so you only have to click OK and the new wanted folder appears in the current folder. Being browser-based, it works equally well locally or remotely.One more good thing about SVN - nothing is ever deleted.