Had some fun with true RNG - Page 1
Forum section: electronics / general

author: Stebanoid
author_rank: Contributor
timestamp: March 24, 2016, 07:33:03 pm
content: Today have had some fun with a true random number generator my colleagues made.It consists of an analog circuit generating random bits and some approved byFSSprocessing to scramble this bits even more. Then it gives these bits to a PC by chunks of 4k or so.It turns out that all that chunks of random data coming from RNG have the same CRC32It's proved that CRC32 calculation is correct. All data coming from RNG are different.<tin foil cap emoticon>Today I was asked to help to solve that puzzle and I did it. O0Ideas?The correct answer will be announced tomorrow.

author: Mechanical Menace
author_rank: Super Contributor
timestamp: March 24, 2016, 07:45:21 pm
content: "All data coming from RNG are different."Probably not a true RNG. 0123456789 followed by 0123456789 can be random. Never seeing it probably isn't. But could be.How can you ever prove you've got a true RNG?

author: Stebanoid
author_rank: Contributor
timestamp: March 24, 2016, 08:06:42 pm
content: You can'tprovethat they are random. But there is a theoretical model what is going on in the noise generator, then this noise is tested by different tests approved or provided by FSS, then it scrambled by ciphering algorithm.This noise passes all tests and the data really look random.

author: AndyC_772
author_rank: Super Contributor
timestamp: March 24, 2016, 08:21:48 pm
content: What's the generator polynomial for your CRC?Thealgorithmcould still technically be OK in the purest mathematical sense, even if your polynomial is all zeroes.The CRC results weren't all zero, by any chance?

author: Stebanoid
author_rank: Contributor
timestamp: March 24, 2016, 08:32:29 pm
content: QuoteI don't remember. Provided by some standard.QuoteNo, they weren't.BTW if I manually change anything in this data, CRC changes too. But what comes out of RNG always have the same CRC.

author: ataradov
author_rank: Super Contributor
timestamp: March 24, 2016, 08:35:10 pm
content: I would guess that it is some property of this FSS approved whitening algorithm.

author: Kilrah
author_rank: Unknown
timestamp: March 24, 2016, 09:02:14 pm
content: Quote from: Stebanoid on March 24, 2016, 07:33:03 pmYou just found why they "approved" that "scrambling" method - because it would actuallyremoverandomness and thus allow them to intercept the data

author: dannyf
author_rank: Super Contributor
timestamp: March 24, 2016, 10:05:57 pm
content: "scramble this bits even more."How do you scramble it more random than random already is?

author: ataradov
author_rank: Super Contributor
timestamp: March 24, 2016, 10:19:09 pm
content: Quote from: dannyf on March 24, 2016, 10:05:57 pmOften TRNGs are not generating a while noise, so a whitening pass is applied to the random stream. I guess that's what OP meant.Although the whole problem seems fishy, someone is obviously doing something wrong and we clearly don't have enough information.

author: Marco
author_rank: Super Contributor
timestamp: March 24, 2016, 10:35:30 pm
content: Hash functions for whitening seem common, so yeah it's almost certainly a result of that.Personally I'd say screw it, use the Neumann method (maybe with some less secure method as backup if the RNG gods screw you and you really need the bits ASAP). Everything else weakens your randomness.

author: Kjelt
author_rank: Super Contributor
timestamp: March 24, 2016, 10:42:05 pm
content: My guess, a crc is a fixed polynomial using the same bit positions, or taps each time. If the random data passes the (statistical) tests then these fixed positions/taps should also be random meaning equal 0 and 1 on those positions, resulting in the same crc result.

author: Red Squirrel
author_rank: Super Contributor
timestamp: March 24, 2016, 10:49:08 pm
content: It's crossed my mind to make a true RNG for my game server, it would be totally pointless, but it would just be cool to be able to say that the drop rates are REALLY random. Could you not just have a microcontroller with a bunch of floating inputs as well as various other sensors like wind speed and light intensity and just use that as data? Or is that still not random enough? Of course you'd want it to have a buffer that it just keeps continually fill so when you request a random number it just grabs from the buffer. First in first out type deal.

author: Stebanoid
author_rank: Contributor
timestamp: March 25, 2016, 05:28:33 am
content: Quote from: dannyf onTodayat 03:05:57Data coming from a physical noise source have someProbability density functionwhich is not flat and bits can have somecorrelationbetween them (it doesn't mean that they are not random). That's why you should apply come whitening function to given data so that you haveuniform distributionof uncorrelated bits.Quote from: Red Squirrel onTodayat 03:49:08It will be used in apublic key infrastructureand your approach neither random enough nor productive enough for that purpose. Such kind of TRNG should provide noise of very high quality and must be tamperproof, so nobody could use a radio noise source near it to change distribution of our generator for example.

author: TerraHertz
author_rank: Super Contributor
timestamp: March 25, 2016, 06:57:53 am
content: Quote from: Stebanoid on March 24, 2016, 07:33:03 pmMy guess: The generator deliberately creates chunks with a standard CRC, to allow easy checking that the random data sequences have not been corrupted.The CRC uniformity could be achieved by just stopping adding data to the chunk when it happens to hit the right CRC.Or more likely, there's an added tail block (or head) that is marked as such, but contains data to fudge the CRC of the entire packet to the standard value.So, no paradox. The data really is random, but the CRC is not. Because the scope of the CRC includes more than the random data. Deliberately.Edit to add: Long ago (back in the 80s) when I was writing gambling games, the very simple hardware used had no built-in RNG. So, I used a software implementation of the old shift register with various taps XOR'd back to the input. The repeating sequence this generates obviously isn't random, but it is quite long, and I checked that all 256 possible byte values were present in the output in quite even weighting. That ran all the time as a constantly active task. To get a true random sequence, I used the best non-determinate source I had - the player's key press timing. Every time they pressed a key, the current shift reg output byte was sampled and shoved in the final random number queue.Other tasks fetched random numbers as required from that queue, and there was no way the user could usefully influence the numbers. Apart from being essential to their generation.

author: Kjelt
author_rank: Super Contributor
timestamp: March 25, 2016, 08:30:38 am
content: Quote from: Stebanoid on March 25, 2016, 05:28:33 amIn this case the source is not good enough for entropy and examining and filtering of those bits that are good enough should take prevalence. Applying a whitening function in my opinion is formatting crypto secure random data to statistical random data and can not be used in secure applications since the whitening function can have problems such as you now had. If it was not for an equal crc there could be some other mathematical dependency found which makes it weak.

author: Stebanoid
author_rank: Contributor
timestamp: March 25, 2016, 08:56:29 am
content: Quote from: Kjelt on March 25, 2016, 08:30:38 amUnfortunately, we live in the world which is not ideal . All sources have some problems. And filtering doesn't help because as Dilbert comics suggests you cannot be sure which number is random and which is not.On the other hand, if you have numbers with uniform distribution and apply x2function to them you'll have another distribution Y. If you take numbers with distribution Y and apply square root function to them you'll have uniformly distributed numbers and they are not worse then the original numbers. And the same with correlation. So, whitening makes sense and necessary in most cases.

author: Kjelt
author_rank: Super Contributor
timestamp: March 25, 2016, 10:12:57 am
content: With filtering I mean to say to only use those bits of the entropy that passes the nist tests.It takes about 80MB of data to test which can take days to collect from a slow entropy source and selecting the correct bits can take months, sometimes concluding it is not good.

author: dannyf
author_rank: Super Contributor
timestamp: March 25, 2016, 12:29:56 pm
content: QuoteIf those truly random bits are auto-correlated, they are not truly random, as the state of the bits now tell you about the future bits.

author: dannyf
author_rank: Super Contributor
timestamp: March 25, 2016, 12:33:31 pm
content: QuoteI did this recently as a way to document some of the experiments I have been doing over the years:https://dannyelectronics.wordpress.com/2016/03/19/true-random-number-generators/It is a mcu + a rc network.The adc approach also works - Kjelt and I had a discussion on that way back - and it has the advantage of an even lower part count: just the mcu,

author: Stebanoid
author_rank: Contributor
timestamp: March 25, 2016, 01:03:54 pm
content: Quote from: dannyf on March 25, 2016, 12:29:56 pmIt's a common misconception. If you roll two dices, the chance to hit 7 is much more then chance to hit 2. I doesn't make them "not random". If you apply appropriate processing to data from dices you can get some amount of uniformly distributed random numbers. And the same with correlation - if you can guess the next bit with probability>0.5 it doesn't mean that you cannot get entropy from that bit (as long as the probability/=1)

author: dannyf
author_rank: Super Contributor
timestamp: March 25, 2016, 01:18:30 pm
content: QuoteTwo different questions.QuoteIt doesn't mean you cannot get entropy from it. Just not as much: the amount of information decreases, in an equal fashion, as the probability approaches 1 or 0.It is hard to have a true random number generator that generates auto-correlated data.

author: Stebanoid
author_rank: Contributor
timestamp: March 25, 2016, 01:25:36 pm
content: Quote from: dannyf on March 25, 2016, 01:18:30 pmSure. You will have fewer random bits at the output than the number of autocorrelated bits at the input. (As it is with dice also...)Quote from: dannyf on March 25, 2016, 01:18:30 pmIt is hard to have a true random number generator.

author: Stebanoid
author_rank: Contributor
timestamp: March 25, 2016, 06:07:24 pm
content: The answer.When I checked that any change in given data changes the CRC, despite it always the same for all data examples, I began to think what is there so deeply related to this polynomial in that data.Maybe this data already contains CRC and when you feed it to the algorithm, it cancels what was calculated before?I phoned to a colleague who made this RNG and yes, there are CRC fields already attached to the end of each piece of several bytes of the data in that big chunk.And this CRC are calculated with the same polynomial.So, it's useful to know that if you calculate CRC(A|CRC(A)) you are going to have the same result for all A.TerraHertz, was very close to the answer!

author: ataradov
author_rank: Super Contributor
timestamp: March 25, 2016, 06:23:21 pm
content: Quote from: Stebanoid on March 25, 2016, 06:07:24 pmNot only it is the same result, it should always be 0. That's the property of CRC and I use this all the time. If I have some protocol where CRC is appended to the data, then I do a simple CRC over the entire array and compare it with 0.Edit: it is going to be 0 provided that bytes of the CRC are stored in a correct order.Edit1: It is also a very intuitive result. CRC is a remainder of the polynomial division, by adding that remainder to the original polynomial you make it divisible by generator polynomial without a remainder.

author: TerraHertz
author_rank: Super Contributor
timestamp: March 29, 2016, 12:22:03 pm
content: Cool. So in fact that method not only allows easy checking if there is data corruption, but allows finding and discarding the damaged sub-chunk.Stebanoid: "So, it's useful to know that if you calculate CRC(A|CRC(A)) you are going to have the same result for all A."ataradov: "Edit: it is going to be 0 provided that bytes of the CRC are stored in a correct order."Presumably those data blocks have headers and other stuff, hence the non-zero CRC.This is interesting to know, and might come in handy.Next: Having a way to force a CRC to zero is neat. But can anyone think of a way to take a block of data, and by adding a few more bytes force the overall CRC to any desired value? (I suppose the ability to do this this would be dependent on the particular CRC algorithm used.)

author: ataradov
author_rank: Super Contributor
timestamp: March 29, 2016, 04:58:35 pm
content: Quote from: TerraHertz on March 29, 2016, 12:22:03 pmIt should be possible to do this based on the same polynomial math. In theory you will need the same number of bytes (bits) as in the generator polynomial.There is really only one CRC algorithm. Specific implementations differ based on the used polynomial value and length, but they all implement the same thing in the end.