Xilinx FPGA bitstream - Page 1
Forum section: electronics / general

author: Unixon
author_rank: Frequent Contributor
timestamp: September 13, 2014, 07:16:14 pm
content: Does anybody know the specs for the bitstream format of Xilinx FPGAs (officially published or obtained by reverse engineering)?In particular, I'm interested in the old Spartan-II family.I know I don't need these specs to simply program the FPGA as I can use the freely available ISE WebPack software on a PC,but my greater interest is in autonomous reprogramming of FPGAs by a custom software running on a SoC/uC/whatever.Can't clearly tell the exact purpose of this approach at the moment, I'm just looking for all possibilities.

author: jancumps
author_rank: Unknown
timestamp: September 13, 2014, 07:25:34 pm
content: I believe that that format is unpublished.All the open source Xilinx related projects that I've checked all rely on the Xilinx propriarity toolchain to synthesise and create the bitfile.

author: miguelvp
author_rank: Super Contributor
timestamp: September 13, 2014, 07:47:56 pm
content: These are relevant for the programming part:http://www.cypress.com/?rID=75048http://www.cypress.com/?rID=46029Also there is a lot of information about doing this without the EZ-USB FX2/FX3 at Xilinx.http://www.xilinx.com/support/documentation/sw_manuals/xilinx11/platform_studio/ps_c_dld_downloading_bitstreams_fpga.htmhttp://www.xilinx.com/support/documentation/sw_manuals/xilinx12_1/platform_studio/ps_c_dld_fpga_configuration_overview.htmAs for bitstream a simple search "xilinx bitstream format" returned a lot of hits including this talk from Milkymist with links to more information about reverse engineering attempts.http://lekernel.net/fpga_toolchain_talk.pdf

author: Balaur
author_rank: Unknown
timestamp: September 13, 2014, 09:54:12 pm
content: Do you:a) only need to configure the device with an existing bitstreamorb) make your own bitstream?If a), then the configuration chapter of the Spartan II Family Datasheet is very informative. It's really easy.If b), then I assume you really have a good reason to do this and therefore, you should be able to invest significant skill and efforts. The works in the research community (especially on topics related to design validation and fault injection through emulation) may be relevant to your interests.

author: Unixon
author_rank: Frequent Contributor
timestamp: September 13, 2014, 10:49:39 pm
content: (b) - make my own bitstreamWell, yes, I want to do a kind of research on experimental processor architectures, some of which may require dynamical reconfiguration which in turn may include self-rewriting (e.g. by a soft-core CPU) as an extreme idea. While many things can be done purely in simulation, I'm looking forward to utilize real hardware for proof-of-concept implementations and, probably, if I ever make it to this point, for practical use.I'm planning to start with Spartan-II/3/3E families, and Spartan-II in particular at first for two reasons: (a) they are cheap enough, so I can experiment with them without having a big problem if I destroy some of them in the process and (b) they should be simpler than modern series.

author: Balaur
author_rank: Unknown
timestamp: September 13, 2014, 11:28:30 pm
content: Good luck!There are many interesting resources. Particularly, thisthesishas a few interesting chapters.

author: bingo600
author_rank: Super Contributor
timestamp: September 14, 2014, 08:37:01 am
content: Collin Oflynn's lubloaderhttp://sourceforge.net/projects/lubloader/xsvf loadershttp://www.clifford.at/libxsvf/http://www.simtec.co.uk/products/EB675001DIP/files/playxsvf-book/book_playxsvf.chapter_generating_xsvf.htmlhttp://dangerousprototypes.com/tag/xsvf/http://dangerousprototypes.com/2012/02/01/jtagwhisperer-an-arduino-jtag-programing-library//Bingo

author: miguelvp
author_rank: Super Contributor
timestamp: September 14, 2014, 08:53:38 am
content: Quote from: bingo600 on September 14, 2014, 08:37:01 amHe is looking for the bitstream format more than for how to program the xilinx.

author: nctnico
author_rank: Super Contributor
timestamp: September 14, 2014, 09:31:38 am
content: Quote from: Unixon on September 13, 2014, 07:16:14 pmMy advise: don't go that route! Just use the serial bitstream interface. It is easy to interface with an SPI interface. In one of my designs I used an SPI flash in which I reserved an area for the FPGA bitstream. By using some extra logic chips I could let the stream from the SPI flash end up in the serial input of the FPGA. All it took to program the FPGA is reading the SPI flash as fast as possible.I implemented JTAG programming a Xilinx FPGA (Spartan II / IIE) from an MCU over a decade ago and that project was a real nightmare. The documentation which exists is wrong so you need to do a lot of experimenting. I actually managed to destroy an FPGA through software!

author: bingo600
author_rank: Super Contributor
timestamp: September 14, 2014, 10:05:03 am
content: Quote from: miguelvp on September 14, 2014, 08:53:38 amAhhh@OP have fun solving that one.... But why ?/Bingo

author: Unixon
author_rank: Frequent Contributor
timestamp: September 14, 2014, 10:16:32 am
content: nctnico, the major obstacle is not the programming an FPGA with a prepared bitstream, but producing a correct bitstream without ISE and not even on a PC. In other words, I want to configure FPGA with my software, not a Xilinx toolchain, because it simply may not exist where I intend to produce FPGA configuration.If an open source FPGA architecture would be available off-the-shelf I wouldn't even bother messing around with Xilinx or Altera or whatever commercial FPGAs out there.

author: miguelvp
author_rank: Super Contributor
timestamp: September 14, 2014, 10:25:04 am
content: Unixon, best of luck to you,The thing is that as a company you have to understand that they want their customers to feel safe with their investments without worries.That's what keeps their employees and their families alive, so go the milkymist route and ignore that corporations are more about their workforce than the corporation itself.But anyways, take a crack at it. You'll learn regardless if you succeed or fail.

author: Unixon
author_rank: Frequent Contributor
timestamp: September 14, 2014, 10:34:24 am
content: Quote from: miguelvp on September 14, 2014, 10:25:04 amThat doesn't explain all of it. Why do they need to keep specs closed? They're just selling chips anyway. Why can't they just sell chips with open specs as other companies do?

author: miguelvp
author_rank: Super Contributor
timestamp: September 14, 2014, 10:40:08 am
content: Quote from: Unixon on September 14, 2014, 10:34:24 amWhat other companies do? not in their field and even so, it's their prerogative to do so.If they want to provide a safe environment it's their call. You can always make your own thing and compete if you think that's the best way to proceed.But a buyer of techonogy might go with them because there is less risk being a closed spec.BTW I'm not really defending them, just putting their shoes in my feet.

author: Unixon
author_rank: Frequent Contributor
timestamp: September 14, 2014, 10:57:13 am
content: Quote from: miguelvp on September 14, 2014, 10:40:08 amI mean well... if you buy an Intel CPU or a micro-controller like AVR you also obtain a number of comprehensive datasheets which explain many things up to the assembly language and machine code of that unit so you can write your compiler to program this unit. Well, we do have the avr-gcc after all. But when it comes to FPGAs for some weird reason FPGA developer/manufacturer does not allow you to have tools to program these chips. What's the big difference? Why is it so important to keep low level architecture at secret?

author: mikeselectricstuff
author_rank: Super Contributor
timestamp: September 14, 2014, 11:10:24 am
content: Quote from: Unixon on September 14, 2014, 10:57:13 amBecause they'd have to document it, and it would make it harder for them to make changes. As it is, if they want to improve the silicon, they make any corresponding changes to their software and the end-users is mostly unaffected.The vast majority of users just don't need these details, so why put in the effort to document it externally.The process of putting a design in an FPGA is hugely more complicated than writing software, and way beyond the ability of the vast majority of end-users.If it's done wrong, it may appear their chips are at fault, which would be a major headache. ISTR reading that a bad configuraction can actually destroy a chip due to excessive power dissipation.And they don't want people to be able to reverse-engineer their customers' designs from the bitstream.From the manufacturers' point of view there is simply no good reason to make it public, and plenty of reasons not to.

author: nctnico
author_rank: Super Contributor
timestamp: September 14, 2014, 11:28:56 am
content: Quote from: Unixon on September 14, 2014, 10:16:32 amHow about creating your own programmable logic array on top of an FPGA? It will be slow but for a proof-of-concept that shouldn't be a problem. If you start with relatively high level blocks (adders, counters, multiplexers/demultiplexers, etc) you could avoid having to deal with boolean logic implementation & optimisation.Alternatively there is a lot of talk on this forum about Psoc devices from Cypress which have run time reconfigurable logic.

author: Unixon
author_rank: Frequent Contributor
timestamp: September 14, 2014, 12:01:25 pm
content: Quote from: nctnico on September 14, 2014, 11:28:56 amYes, I was thinking about that for a while and decided to use this option if I can't do anything better.

author: nctnico
author_rank: Super Contributor
timestamp: September 14, 2014, 12:20:52 pm
content: There are probably papers on this subject as well but in the past I have made more or less software configurable FPGA designs. In those designs (also on Spartan II) I used a bi-directional bus (not supported in Spartan 3 and later!) and used a block ram to store the program. Each step would read data from one functional unit and write it into another functional unit.

author: miguelvp
author_rank: Super Contributor
timestamp: September 14, 2014, 06:05:14 pm
content: Say I'm a manufacturer let's say Rigol for example that use Spartan 6 FPGAs in their products 3 of them in the 4000 4 channel series. Why will i choose Xilinx again if their bitstream or worse their encryption has been compromised.Sometimes reverse engineering can have adverse consequences to the manufacturer.The difference with a CPU is that it holds no code (and really neither does an FPGA since it programmed at boot time) but a manufacturer has all the right in the world to keep their applications secure so that their clients feel confident on the decision of selecting that manufacturer.They provide you with the tools to program them and if someone had a better tool I bet they might comply after a license agreement or partnership of sorts after the mandatory NDA and legal obligations.So to me I can see why it's not public domain, they do however offer plenty of information about everything we all need other than their bitstream and encryption mechanisms.Again, just playing the devil's advocate here, I'm really not opposed to reverse engineering.Did you look at Milkymists presentation I linked before? It actually tell you what needs to be done but no how to actually do it. Also he links parallel efforts.

author: Unixon
author_rank: Frequent Contributor
timestamp: September 14, 2014, 06:35:06 pm
content: Quote from: miguelvp on September 14, 2014, 06:05:14 pmI don't know what Rigol or other manufacturers rely on, but I know even if I were to launch a product like that I wouldn't bother worrying about keeping secrets safe.My way of thinking tells that "when it's published - it's published, just assume your secrets are already known to everyone, quickly get over it and move on."I doubt the bitstream protection is that much worthwhile: in comparison, a CPU code can be disassembled as well and nobody really cares about that. However, manufacturers try to protect firmware, so it's somewhat similar going on there.Security by obscurity had always been a bad way of doing things in [at least modern] cryptography, but this principle still lives in business models which is very sad.Yes, I browsed the presentation. I think I saw it before when I began to investigate this subject.I though that maybe someone had greater progress in this field, but it seems that this is as far as things went up until now.

author: hamster_nz
author_rank: Super Contributor
timestamp: September 14, 2014, 10:38:34 pm
content: Quote from: Unixon on September 14, 2014, 10:57:13 amAll vendors bend over backwards to help you design for their chips - user guides, reference guides, App notes and so on. There is more documentation than I could ever read...However I think that the bitstream specs are not published for good reason. I don't beleive that FPGA manufactures want to the reliability and performance of their hardware product to be determined by the quality of homebrewed synthsis software. No Open Source project could put the engineering resources, time and effort in that would be required to validate the software to a level that makes it useful for any meaningful commercial development.I wouldn't want my core datacenter swtiches to be running with bitstreams from unvalidated tools. In fact I wouldn't expect anybody to use free tools for any product where the commercial risk exceeded the cost of a vendor toolset license - and that is pretty much any real-world commercial product.

author: legacy
author_rank: Super Contributor
timestamp: September 14, 2014, 11:16:10 pm
content: Quote from: hamster_nz on September 14, 2014, 10:38:34 pmgood point, i think so, too

author: Unixon
author_rank: Frequent Contributor
timestamp: September 15, 2014, 02:49:16 am
content: Quote from: hamster_nz on September 14, 2014, 10:38:34 pmThe reliability of an FPGA is only determined by the reliability of bare silicon, it has nothing to do with bitstream, firmware or whatever other software a product user may want to run on this underlying hardware. Anything that is not hardwired on the die is a software. Speaking that someone's software determines the reliability of someone's else hardware is plain wrong and restricting the software that can run on a particular hardware or even the very ways that software can be produced is weird and surreal.Quote from: hamster_nz on September 14, 2014, 10:38:34 pmThe openness has nothing to do with the quality of a software. I would never trust a tool I can't verify myself and all commercial tools are not verifiable as they're closed, hence all commercial tools are much more untrusted no matter what resources a manufacturer had put in verifying the product, it is virtually zero as I can't verify that myself.Quote from: hamster_nz on September 14, 2014, 10:38:34 pmThen just don't buy products made with tools you don't like, but don't force other people to behave exactly as you do.

author: hamster_nz
author_rank: Super Contributor
timestamp: September 15, 2014, 04:48:45 am
content: Quote from: Unixon on September 15, 2014, 02:49:16 amAre you saying that:1) There is no way that a badly constructed bitstream can cause harm to an FPGA2) That a place and route process can't turn a valid logical design into a design with reliablity issues - for example failed timing for paths, metastability issues, chip-level hot spots, dodgy startup states, clock skew over reset networks, flakeyness due to carry chain length issues, voltage and temp corner cases and so on...Quote from: Unixon on September 15, 2014, 02:49:16 amI can't force anybody to do anything - and likewise you should allow FPGA manufactures to keep their bitstreams closed if they want to.... they don't have to share the results of their R&D with you.

author: Unixon
author_rank: Frequent Contributor
timestamp: September 15, 2014, 07:45:02 am
content: Quote from: hamster_nz on September 15, 2014, 04:48:45 amI'm saying that both of these issues has nothing to do with reliability of an FPGA itself. A hardware designer that puts an incorrect bitstream into FPGA can have reliability problems with his design, but this is unrelated to the FPGA chip. If somebody wants to fry his FPGA with flawed configuration he has all rights to do so.And speaking about the reliability of an FPGA... I wouldn't consider reliable an architecture of FPGA that allows a badly constructed bitstream to cause a malfunction or lead to irreversible changes in the die.Quote from: hamster_nz on September 15, 2014, 04:48:45 amI don't mind them having protection mechanisms, including encryption, that make it impossible to readout a customer's configuration - that's totally OK, I appreciate that, but closing the architecture specs for this purpose is an overkill.

author: andersm
author_rank: Super Contributor
timestamp: September 15, 2014, 08:16:05 am
content: ISTR reading that a long time ago, Xilinx did publish the bitstream format for one of their devices for use in universities. However in the end this produced very little results, and they stopped the practice because it just wasn't worth the effort.Have you reviewed the existing research on the subject (eg.http://www.isi.edu/~nsteiner/publications/soni-2013-bitstream-fccm13.pdf)?

author: Unixon
author_rank: Frequent Contributor
timestamp: September 15, 2014, 08:28:05 am
content: Quote from: andersm on September 15, 2014, 08:16:05 amOh, thanks. Haven't seen this paper.Quickly browsed through pages... the theory look OK, but I didn't get it clearly from a first glance to what degree this is applicable to real FPGAs I intend to experiment with. Have to read this slowly and carefully

author: mikeselectricstuff
author_rank: Super Contributor
timestamp: September 15, 2014, 08:56:08 am
content: Quote from: Unixon on September 15, 2014, 02:49:16 amRubbish.It also depends heavily on the accuracy of the timing model. A toolchain that doesn't get this right could easily produce designs that are marginal and risk failing over temperature or part-to-part variation

author: mikeselectricstuff
author_rank: Super Contributor
timestamp: September 15, 2014, 08:59:26 am
content: QuoteThe FPGA toolchain is too complex for pretty much anyone to understand, let alone verify.If you don't trust anything you can't verify yourself then you'll just have to live in the stone age and not use any modern technology.

author: Unixon
author_rank: Frequent Contributor
timestamp: September 15, 2014, 09:50:43 am
content: Quote from: mikeselectricstuff on September 15, 2014, 08:56:08 amThis is true in respect of a particular design, not the FPGA. A poor design may fail for any reason, but the FPGA doesn't care if it does, it will just process signals under provided configuration.Quote from: mikeselectricstuff on September 15, 2014, 08:59:26 amNope. This only means you don't have to trust in something to use it.

author: Balaur
author_rank: Unknown
timestamp: September 15, 2014, 02:25:35 pm
content: Quote from: Unixon on September 15, 2014, 09:50:43 amThis is not what Mike said.The toolchain (especially on the synthesis and timing analysis part) will have to meet timing constraints that you may have set. This happens through logic, placement and routing choices and optimization that are obviously dependent on 1) the FPGA architecture and 2) the physical characterization data that the tools use for the timing/power analysis.For the overall process to be successful, several requirements must be met:- the synthesis is successful, i.e. your synthesis tool managed to translate some kind of High-Level Synthesis (HLS) design representation - it may be RTL or block-level schematic or whatever to the exact LUT/CLB/VersaTile your FPGA may have. This is not trivial- the placement of the cells and the routing of the signals is successful- if you have a time-constrained design (i.e. it needs to work at a given frequency), you will have to trust that the tools from the design flow managed to fulfill your constraints. Failing that, you need to look at the report from timing analysis tools to work around the actual performance you get. This is the part that Mike discussed. If the tools are not good or if the primary silicon data they use is not accurate, their analysis may be wrong. Your design may work, almost work at 156.25 MHz required for your 10Gb Ethernet or fail miserably when some particular condition is met, causing a lot of headaches.

author: Unixon
author_rank: Frequent Contributor
timestamp: September 15, 2014, 03:34:55 pm
content: We argued mostly about the terminology. I understand what you mean, but this is related to the reliability of a design, not the reliability of an FPGA. I agree, that a toolchain that doesn't do project verification well enough may produce poor configuration even from a perfectly designed project sources. My point is that while these issues may affect the reliability of a design and a product where that design goes to, under no circumstances we can say that it affects the reliability of the FPGA we used since the FPGA chip itself doesn't care what we put into it.Even if a trusted toolchain is used, we can't tell that it OK, unless the product is tested under all possible conditions.My other point was that since it is the product testing procedures that decide if it works as intended or not, we may use toolchains of any level of trust, those that fail will just drop out of competition.

author: mikeselectricstuff
author_rank: Super Contributor
timestamp: September 15, 2014, 03:58:23 pm
content: Quote from: Unixon on September 15, 2014, 03:34:55 pmWhat's the difference?It is impossible to use an FPGA without design software. If that design software does not accurately know about the timing behaviour of every part of the device, it can produce a bitstream that doesn't work reliably, and you won't necessarily know until you start seeing field failures, or designs that fail on some chips but not others . This is to all intents and purposes indistinguishable from bad hardware.This is a completely different scenario to a processor, which if clocked within spec is guaranteed to execute any code under all circumstances.

author: Unixon
author_rank: Frequent Contributor
timestamp: September 15, 2014, 04:16:32 pm
content: Quote from: mikeselectricstuff on September 15, 2014, 03:58:23 pmIf an underlying FPGA fabric is bad, you are to blame the manufacturer of that FPGA chip.If a toolchain is bad, you are to blame the developer of that toolchain.If a design is bad, you are to blame the designer that made it.While these may become indistinguishable in a hardware, it is not a good practice to blame ones for other's faults.

author: Bassman59
author_rank: Super Contributor
timestamp: September 15, 2014, 04:26:44 pm
content: Quote from: Unixon on September 14, 2014, 10:34:24 amBecause they don't want to deal with the support issues, nor the liability issues involved with a customer loading a bad bitstream.And anyways, none of the FPGA vendors have published their bitstream specs.

author: Bassman59
author_rank: Super Contributor
timestamp: September 15, 2014, 04:39:54 pm
content: Quote from: Unixon on September 14, 2014, 10:16:32 amThat's never going to happen. Unless there's some compelling architectural improvement, attracting the investment capital necessary to bring the parts into production won't happen. The incumbents are just too entrenched. And an open bitstream is not even on the list of things that matter to most customers.

author: Bassman59
author_rank: Super Contributor
timestamp: September 15, 2014, 04:49:29 pm
content: Quote from: andersm on September 15, 2014, 08:16:05 amXC6200, long since discontinued.

author: mikeselectricstuff
author_rank: Super Contributor
timestamp: September 15, 2014, 05:35:03 pm
content: Quote from: Unixon on September 15, 2014, 04:16:32 pmThe problem is that "bad" is not a binary condition. I have no doubt that FPGA toolchains include all sorts of tweaks and fudges to work around or improve issues that are actually silicon related.If the silicon vendor supplies the tools, there is one point of blame if there's a problem, as soon as you seperate them, it could get very difficult to figure out who's to blame.

author: abyrvalg
author_rank: Frequent Contributor
timestamp: September 15, 2014, 06:12:01 pm
content: Does existense of GCC affect reliability of CPUs in general? What are you guys talking about? If you need reliability - you buy something like Intel C Compiler, ARM Developer Suite (or even something "hardened" from Wind River). If you don't care and just want to go cheap - use GCC (that's another question - how does GCC's quality compares to those commercial's). That's "free market", no?There are so many other possible points of FPGA failure - cheap unexperienced HDL engineers, cheap PCB designers, fake voltage regulators powering the board etc, why then don't force using all these from FPGA manufacturer too? - this will increase the Holy Reliability! Let's go extreme: don't publish the tools at all, don't publish the pinouts - voila! you'll have a single totally reliable (according to your logic) source of FPGA-based products.Unixon, check this also:https://code.google.com/p/debit/

author: marshallh
author_rank: Unknown
timestamp: September 15, 2014, 06:23:51 pm
content: Friend of mine has reversed the Coolrunner II architecture. It's a simple CPLD. Let me know if you want more info.But really, there is no point in whinging about no open source alternative, when the OSS guys themselves tell us closed source fools "if you want it so bad just make it yourself".