Package: lytest
lytestTest automation tools for integrated circuit layout using klayout and pytest and for PCBs using KiCAD, pcbnew, andkigadgets.Code testingWe don't know by inspection what code does. Determining its behavior involves running it. Is this behavior what we want? There are three basic questions:Does the code runIs its behavior correctHas its behavior changedIdeally, we want all of these questions answered precisely for a whole range of tests immediately after any code change. But code changes constantly. This issue is excellently addressed bypytest, an automated unit testing framework. In a single command, it scrolls through a bunch of test functions, makes sure they run, and makes some programmer-defined checks on behavior.Code for layoutCode for layout is different from regular code in that its behavior is the geometry it produces. It is difficult to state as text what the correct behavior is, so layouts must be reviewed by eye. This process takes a lot of time for even one complex layout; it is only as good as the reviewer's eyes and knowledge; and it cannot practically be done without hundreds of commits since the last review (from multiple collaborators), making it very hard to localize the origin of bugs.What lytest doeslytestaddresses the layout behavior testing problem by fully automating a key part of layout review process: change detection. It combines thepytestautomated testing framework with theklayoutXOR differencing engine. If stored GDS reference files are deemed correct, then change detection is as good as answering the question of correctness.A test consists of a fixed block of code that produces a GDSII (or OASIS) file. An initial run produces a reference layout. After review by a human, this file is then marked as the "correct" behavior for that block of code. When the tests are executed, the block runs again, producing a new "run" GDS. Differences in geometry (i.e. non-empty XORs) will raise an exception to the attention of whoever is conducting the test.Installationpip install lytestlytesthas several optional dependencies. It does not install layout packages (phidl,klayout, etc.). At least one of them must be installed somehow: you pick how. If usingklayout, a.k.a.pya, alsopip install lygadgets.lygadgetsmanages klayout's environment.lytesthas the ability to send layout diffs from the testing scripts to the KLayout GUI. To enable this functionality,pip install lyipc && lygadgets_link lyipcUsageThere are three main parts: write the test, save the answer, run the test repeatedly.Write a testA simple code test that is compatible withpytestlooks like this.deftest_addition():assert1+1==2There are two magic parts oflytestthat make XOR testing about as simple: layout containers likecontained_XXXand the XOR test decorator:difftest_it. Here is a complete test file in which we test thewaveguidedevice frommy_library(phidl language version).fromlytestimportcontained_phidlDevice,difftest_itimportmy_libraryaslib@contained_phidlDevicedefBasicWaveguides(TOP):TOP.add_reference(lib.waveguide(width=0.5,length=20))deftest_BasicWaveguides():difftest_it(BasicWaveguides)()The "contained" function (BasicWaveguides) isnota pytest function. It takes an (empty) cell, modifies that cell, and returns nothing. Optional arguments are allowed. They cannot start with "test_" or end with "_test". There is a bit of a magic associated with declaring and debugging contained layouts, discussed below. For now, just go with that way of thinking about it.The pytest function (test_BasicWaveguides) is essentially just a renaming ofdifftest_itwrapping your contained geometry function. Difftest it implements compiling the test layout, the XOR test, and error reporting. All of these second functions have the exact same format, which is why they are written as one-liners. If you want to disable a test from running automatically with pytest, just comment out this second function.Why two functions? The first is a normal-ish function. It can be called, examined, used to save to file. It is useful beyond being a test (see below). The second is run automatically and has a whole bunch of other things happening, such as the XOR testing itself.Making it a better testPut in a few permutations of arguments. Check corner cases. Maybe intentionally break it usingpytest.raises.defBasicWaveguides(TOP):TOP.add_reference(lib.waveguide(width=0.5,length=20))wg2=lib.waveguide(width=0.5,length=50)TOP.add_reference(wg2.rotate(90))withpytest.raises(ValueError):wg3=lib.waveguide(width=-0.5,length=50)Save the answerReference layouts are stored in the "ref_layouts" directory. The first time you run a new test, it will put its result in ref_layouts, where it will be fixed. If that code changes and you run test again, it will raise a geometry difference error. So if you deem the new behavior to be correct, update the reference. This is done from command line:lyteststoretest_geometries.pyBasicWaveguidesreplacing "test_geometries.py" with the filename and "BasicWaveguides" with whatever yours is called.OASIS (optional)This model necessitates tracking references layouts. They are large binaries that change often, which is a bad combination for version control schemes. The OASIS format is much more memory efficient. It is supported since v0.0.4 and can be selected in thedifftest_itcall (see klayout examples).Run the testThe terminal commandpytest [target]will run the file target (a .py file). If target is a directory, it crawls through automatically running .py files that start withtest_or end with_test. Within a file, pytest automatically calls every function starting withtest_or ending with_test. If an exception is raised, it prints the stack trace but keeps going on to the next function.You can also pick out a single test and run it withlytestruntest_geometries.pyBasicWaveguidesVisualizing errors (optional)lyipcstands for klayout inter-process control, but it is essentially a visual debug tool.lytestandlyipcare designed to work together to give more visual information. During development, the contained geometry result is sent automatically to the GUI. During testing, failed tests are prepped for XOR in the GUI. Get it through the klayout salt Package ManagerContinuous Integration (optional)Continuous integration (CI) is when tests are run in an automated way in connection with a version control system. Every time a push is made to any branch, the branch is pulled into a virtual machine (located at travis-ci.org), and a predefined test suite is run. After a few minutes, github displays whether that branch is passing. lytest has CI and an example of how to set it up can be found in.travis.yml. Since klayout standalone takes a long time to build, it is recommended to turn on the caching option for pip.Supported formatsIntegrated circuit files end with ".gds" or ".oas" and are handled byphidlandpya. KiCAD PCBs end with ".kicad_pcb" and are handled bykigadgets. The container anddifftest_itpattern used above work exactly the same with these exceptions:kigadgetshas no equivalent oflyipc, so the automatic application integration does not happen.kigadgetsusing a hashing approach to equivalence instead of KLayout's XOR method. There is no built-in way to view what is different between .kicad_pcb's, but an external program would be used for more detail: ex.kicad-diff.git integrationGDS and OASIS are binary formats, so they cannot be compared meaningfully by typical text diffs. This is really an issue with version control approaches because they rely on diffing across commits, staging areas, and branches.lytesteffectively gives a way to diff layouts, so it can help. It only matters if it pops up the two files in klayout, so you need to havelyipcserver active. This is cool, trust me.SetupYou need to configure your own git system to enable it. This is simply done bylytestgit-configYou can also do this project-by-project using the--localflag.And then you can do things likegitdifffeature-branchtests/ref_layouts/to see all the differences go over to the klayout GUI.The lytest/lyipc/ipython test-driven workflowI currently use this workflow when developing new device cells (as opposed to system-level cells - a different workflow). It is a graphical layout version of test-driven design. It is enabled by some of the tools in lytest.Whenever you write a new function, you call it with various options in order to develop it and understand what its doing. You come up with a mixture of library behavior and usgage calls that you like. If you save those calls in the right place, you have made atest!Tools and setuplyipc(klayout inter-process control)like quickplot for the klayout window, hence "kqp"kqp sends intermediate layouts at run time from the debugger to the klayout GUIlyipc is also used to automatically bring up failed XOR tests so that XOR results can be visualizedpdb(python debugger)terminal-based, as lightweight as it getsset breakpoint withimport pdb; pdb.set_trace()you can call quickplot or kqp (klayout_quickplot) from itrecommended:pdb++ipython (interactive python)an upgraded command line shellclose interaction with changing code, autoreloadingturn autoreload on by default by puttingprint('autoreload is on')%load_extautoreload%autoreload2in~/.ipython/profile_default/startup/auto_reloader.ipyThe processLet's say you want to design a qubit. After setting out your goals on paper, start by making a container function with some basic behavior# test_qubits.pyfromlytestimportcontained_phidlDevice,difftest_itimportmy_libraryaslib@contained_phidlDevicedefSomeQubits(TOP):TOP<<lib.qubit()# def test_SomeQubits(): difftest_it(SomeQubits)()and the corresponding library function# my_library.py...defqubit():D=Device('qubit')# geometry goes herereturnD...Activate the lyIPC server in klayout GUI. Open up an ipython shell (with autoreload on). Add some behavior to the library function. To see what this did,[1]%load_extautoreload[2]%autoreload2[3]fromtest_qubitsimportSomeQubits[4]SomeQubits()voila. Your contained layout has appeared in your klayout GUI.Change the library behavior. Call it again[3]SomeQubits()voila! Thenewbehavior appears. No reloading all of the overhead or dealing with layout boilerplate. This call can be repeated every time you save the library.As you are developing, there might be some call combinations that are very relevant. Keep those. You might end up with# test_qubits.py...@contained_phidlDevicedefSomeQubits(TOP):TOP<<lib.qubit()TOP<<lib.qubit(detuing=100e6).movey(100)TOP<<lib.qubit(local_realistic=True).movex(100)...Finally, when you are satisfied with the library behavior, turn it into a test that is run automatically, telling all your collaborators - hey, don't change anything that ends up breaking theSomeQubitscontainer. To do this, just uncomment the line above that has difftest_it.What is this container thing?A layout "container" appears different from the outside vs. the inside. From the outside, it is a function. It has one optional argument that is a filename. When called, it makes a layout and saves it to that file. From the outside, containers are not language specific. The caller does not know if phidl, pya, or some other geometry language is being used.The container can be used in different ways. Sometimes that layout gets saved to a file, otherwise it is sent for quickplotting or used in a XOR test. The critical thing is that the geometry code inside of the container remains identical no matter how we want to use it.From the inside, the container looks completely different. It appears to receive – not a filename – but python objecs: a Cell (pya) or Device (phidl). So the container is also a wrapper that takes care of the set up and tear down associated with turning geometry commands into a complete layout. Script containers are not pya or phidl specific. They contain arbitrary code that produces a file and must return that filename to be found by the container.Authors: Alex Tait, Adam McCaughan, Sonia Buckley, Jeff Chiles, Jeff Shainline, Rich Mirin, Sae Woo NamNational Institute of Standards and Technology, Boulder, CO, USA