Package: puid-py
PythonpuidSimple, flexible and efficient generation of probably unique identifiers (puid, aka random strings) of intuitively specified entropy using pre-defined or custom characters (including Unicode).frompuidimportChars,Puidrand_id=Puid(chars=Chars.ALPHA,total=1e5,risk=1e12)rand_id.generate()'nWwiLXwdKsTcr'TOCOverviewUsageInstallationAPICharsMotivationWhat is a random string?How random is a random string?UniquenessID randomnessEfficiencyOverkill and Under SpecifyEfficienciestl;drOverviewpuidprovides intuitive and efficient generation of random IDs. For the purposes ofpuid, a random ID is considered a random string used in a context of uniqueness, that is, random IDs are a bunch of random strings that are hopefully unique.Random string generation can be thought of as atransformationof some random source of entropy into a stringrepresentationof randomness. A general purpose random string library used for random IDs should therefore provide user specification for each of the following three key aspects:Entropy sourceWhat source of randomness is being transformed?puidallows easy specification of the function used for source randomnessID charactersWhat characters are used in the ID?puidprovides 19 pre-defined character sets, as well as allows custom characters, including UnicodeID randomnessWhat is the resulting “randomness” of the IDs?puidallows an intuitive, explicit specification of ID randomnessTOCUsageCreating a random ID generator usingpuidis a simple as:frompuidimportPuidrand_id=Puid()rand_id.generate()'a78gWq7N51paZ2Hx5qkoK3'Entropy Sourcepuidusessecrets.token_bytesas the default entropy source. Theentropy_sourceoption can be used to configure a specific entropy source:frompuidimportPuidfromrandomimportgetrandbitsdefprng_bytes(n):returnbytearray(getrandbits(8)for_inrange(n))prng_id=Puid(entropy_source=prng_bytes)prng_id.generate()'JcQTr8u7MATncImOjO0qOS'ID CharactersBy default,puiduse theRFC 4648file system & URL safe characters. Thecharsoption can by used to specify any of 16pre-defined character setsor custom characters, including Unicode:frompuidimportChars,Puidhex_id=Puid(chars=Chars.HEX)hex_id.generate()'a4b130ba638fc7db5d87e064a21e6b46'dingosky_id=Puid(chars='dingosky')dingosky_id.generate()'sdosigokdsdygooggogdggisndkogonksnkodnokosg'unicode_id=Puid(chars='dîñgø$kyDÎÑGØßK¥')unicode_id.generate()'îGÎØÎÑî¥gK¥Ñ¥kîDîyøøØñÑØyd¥¥ØGØÑ$KßØgøÑ'ID RandomnessGenerated IDs have 128-bit entropy by default.puidprovides a simple, intuitive way to specify ID randomness by declaring atotalnumber of possible IDs with a specifiedriskof a repeat in that many IDs:To generate up to10 millionrandom IDs with1 in a trillionchance of repeat:frompuidimportChars,Puidsafe32_id=Puid(total=10e6,risk=1e15,chars=Chars.SAFE32)safe32_id.generate()'d7ntFnH4FngrqgdR3Dtt'Thebitsoption can be used to directly specify an amount of ID randomness:frompuidimportChars,Puidtoken=Puid(bits=256,chars=Chars.HEX_UPPER)token.generate()'5D241826F2A644E1B725DB1DD7E4BF742D9D0DC6D6A36F419046A02835A16B83'TOCInstallationPyPipipinstallpuid-pyCondacondainstall-cdingoskypuid-pyTOCAPIpuidexports the classPuidwhich used to create random ID generators.Puidoptionally takes arguments to configuration ID generation:total: Total number of potential (i.e. expected) IDsrisk: Risk of repeat intotalIDsbits: ID entropy bitschars: ID charactersentropy_source: Function of the form(n: number) => bytearrayfor source entropyNotesAll config fields are optionaltotal/riskmust be set togethertotal/riskandbitscannot both be setcharsmust be valid custom characters or pre-definedCharsentropy_sourceis a function of the form(n: number) => bytearrayDefaultsbits: 128chars:Chars.SAFE64entropy_source:secret.token_bytesPuidInfoThePuid's__repr__function provides information regarding the generator configuration:bits: ID entropybits_per_char: Entropy bits per ID characterchars: Source charactersentropy_source: Stringmodule.functionere: Entropy representation efficiencylen: ID string lengthExample:frompuidimportChars,Puidrand_id=Puid(total=1e5,risk=1e14,chars=Chars.BASE32)rand_id.generate()'7XKJJKNZBF7GCMEX'print(rand_id)Puid:bits=80.0,bits_per_char=5.0,chars=BASE32->'234567ABCDEFGHIJKLMNOPQRSTUVWXYZ',len=16,ere=0.625,entropy_source=secrets.token_bytesCharsThere are 19 pre-defined character sets:NameCharactersALPHAABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzALPHA_LOWERabcdefghijklmnopqrstuvwxyzALPHA_UPPERABCDEFGHIJKLMNOPQRSTUVWXYZALPHANUMABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789ALPHANUM_LOWERabcdefghijklmnopqrstuvwxyz0123456789ALPHANUM_UPPERABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789BASE160123456789ABCDEFBASE32ABCDEFGHIJKLMNOPQRSTUVWXYZ234567BASE32_HEX0123456789abcdefghijklmnopqrstuvBASE32_HEX_UPPER0123456789ABCDEFGHIJKLMNOPQRSTUVCROCKFORD320123456789ABCDEFGHJKMNPQRSTVWXYZDECIMAL0123456789HEX0123456789abcdefHEX_UPPER0123456789ABCDEFSAFE_ASCII!#$%&()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstuvwxyz{|}~SAFE322346789bdfghjmnpqrtBDFGHJLMNPQRTSAFE64ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_SYMBOL!#$%&()*+,-./:;<=>?@[]^_{|}~WORD_SAFE3223456789CFGHJMPQRVWXcfghjmpqrvwxAny string of up to 256 unique characters can be used forpuidgeneration.Description of non-obvious character setsNameDescriptionBASE16https://datatracker.ietf.org/doc/html/rfc4648#section-8BASE32https://datatracker.ietf.org/doc/html/rfc4648#section-6BASE32_HEXLowercase of Base32HexUpperBASE32_HEX_UPPERhttps://datatracker.ietf.org/doc/html/rfc4648#section-7CROCKFORD32https://www.crockford.com/base32.htmlSAFE_ASCIIPrintable ascii that does not require escape in StringSAFE32Alpha and numbers picked to reduce chance of English wordsSAFE64https://datatracker.ietf.org/doc/html/rfc4648#section-5WORD_SAFE32Alpha and numbers picked to reduce chance of English wordsNote: SAFE32 and WORD_SAFE32 are two different strategies for the same goal.TOCMotivationDevelopers frequently need random strings in applications ranging from long-term (e.g., data store keys) to short-term (e.g. DOM IDs on a web page). These IDs are, of course, of secondary concern. No one wants to think about them much, they just want to be easy to generate.But developersshouldthink about the random strings they use. The generation of random IDs is a design choice, and just like any other design choice, that choice should be explicit in nature and based on a familiar with why such choices are made. Yet a cursory review of random string libraries, as well as random string usage in many applications, yields a lack of clarity that belies careful consideration.TOCWhat is a random string?Although this may seem to have an obvious answer, there is actually a key, often overlooked subtlety: a random stringis not randomin and of itself. To understand this, we need to understandentropyas it relates to computers.A somewhat simplistic statement for entropy from information theory is:entropy is a measure of uncertainty in the possible outcomes of an event. Given the base 2 system inherent in computers, this uncertainty naturally maps to a unit of bits (known as Shannon entropy). So we see statements like "this random string has 128 bits of entropy". But here is the subtlety:A random string does not have entropyRather, a random string representscapturedentropy, entropy that was produced bysome otherprocess. For example, you cannot look at the hex string'18f6303a'and definitively say it has 32 bits of entropy. To see why, suppose you run the following code snippet and get'18f6303a':fromrandomimportrandomrand_id=lambda:'18f6303a'ifrandom()<0.5else'1'rand_id()'18f6303a'The entropy of the resulting string'18f6303a'is 1 bit. That's it; 1 bit. The same entropy as when the outcome'1'is observed. In either case, there are two equally possible outcomes and the resulting entropy is therefore 1 bit. It's important to have this clear understanding:Entropy is a measure in the uncertainty of an event, independent of the representation of that uncertaintyIn information theory you would state the random process emits two symbols,18f6303aand1, and the outcome is equally likely to be either symbol. Hence there is 1 bit of entropy in the process. The symbols don't matter. It would be much more likely to see the symbolsTandF, or0and1, or evenONandOFF, but regardless, the processproduces1 bit of entropy and symbols used torepresentthat entropy do not effect the entropy itself.Entropy sourceRandom string generators need an external source of entropy and typically use a system resource for that entropy. In Python, this could be, for example,secrets.token_bytesorrandom.getrandbits. Nonetheless, it is important to appreciate that the properties of the generated random strings depend on the characteristics of the entropy source. Whether a random string is suitable for use as a secure token depends on the security characteristics of the entropy source, not on the string representation of the token.ID charactersAs noted, the characters (symbols) used for a random string do not determine the entropy. However, the number of unique characters available does. Under the assumption that each character is equally probable (which maximizes entropy) it is easy to show the entropy per character is a constant log2(N), whereNis of the number of characters available.ID randomnessString randomness is determined by the entropy per character times the number of characters in the string. Thequalityof that randomness is directly tied to the quality of the entropy source. Therandomnessdepends on the number of available characters and the length of the string.And finally we can state: a random string is a character representation of captured entropy.TOCUniquenessThe goal ofpuidis to provide simple, intuitive random ID generation using random strings. As noted above, we can consider random string generation as thetransformationof system entropy into a characterrepresentation, and random IDs as being the use of such random strings to represent unique IDs. There is a catch though; a big catch:Random strings do not produce unique IDsRecall that entropy is the measure of uncertainty in the possible outcomes of an event. It is critical that the uncertainty of each event isindependentof all prior events. This means two separate eventscanproduce the same result (i.e., the same ID); otherwise the process isn't random. You could, of course, compare each generated random string to all prior IDs and thereby achieve uniqueness. But some such post-processing must occur to ensure random IDs are truly unique.Deterministic uniqueness checks, however, incur significant processing overhead and are rarely used. Instead, developers (knowingly?) relax the requirement that random IDs are truly, deterministically unique for a much lesser standard, one of probabilistic uniqueness. We "trust" that randomly generated IDs are unique by virtue of the chance of a repeated ID being very low.And once again, we reach a point of subtlety. (And we thought random strings were easy!) The "trust" that randomly generated IDs are unique actually turns entropy as it's been discussed thus far on it's head. Instead of viewing entropy as a measure of uncertainty in thegenerationof IDs, we consider entropy as a measure of the probability that no two IDs will be the same. To be sure, we want this probability to be very low, but for random strings itcannot be zero! And to be clear,entropy is not such a measure. Not directly anyway. Yes, the higher the entropy, the lower the probability, but it takes a bit of math to correlate the two in a proper manner. (Don't worry,puidtakes care of this math for you).Furthermore, the probable uniqueness of ID generation is always in some limited context. Consider IDs for a data store. You don't care if a generated ID is the same as an ID used in another data store in another application in another company in a galaxy far, far away. You care that the ID is (probably) unique within the context of your application.To recap, random string generation does not produce unique IDs, but rather, IDs that are probably unique (within some context). That subtlety is important enough it's baked into the name ofpuid. And it is fully at odds with the naming of a version 4uuid. Why? Because being generated via a random process means auuidcannot be unique. As a corollary, it can't be universal either. As noted above, we don't care about the universal part anyway, but the fact remains, auuidisn'tuu.TOCID randomnessSo what does the statement "these IDs have 122 bits of entropy" actually mean? Entropy is a measure of uncertainty after all, and we're concerned that our IDs be unique, probably unique anyway. So what does "122 bits of entropy" mean for the probable uniqueness of IDs?First, let's be clear what itdoesn'tmean. We're concerned with uniqueness of a bunch of IDs in a certain context. The randomness ofany oneof those ID isn't the real concern. Yes, we can say "given 122 bits of entropy" each ID has a probability of 2-122of occurring. And yes, that certainly makes the occurrence of any particular ID rare. But with respect to the uniqueness of IDs, it simply isn't "enough" to tell the whole story.And here again we hit another subtlety. It turns out the question, as posed, is underspecified, i.e. it is not specific enough to be answered. To properly determine how entropy relates to the probable uniqueness of IDs, we need to specifyhow manyIDs are to be generated in a certain context. Only then can we determine the probability of generating unique IDs. So our question really needs to be: givenNbits of entropy, what is the probability of uniqueness inTrandom IDs?Fortunately, there is a mathematical correlation between entropy and the probability of uniqueness. This correlation is often explored via theBirthday Paradox. Why paradox? Because the relationship, when cast as a problem of unique birthdays in some number of people, is initially quite surprising. But nonetheless, the relationship exists, it is well-known, andpuidwill take care of the math for us.At this point we can now note that rather than say "these IDs haveNbits of entropy", we actually want to say "generatingTof these IDs has a riskRof a repeat". And fortunately,puidallows straightforward specification of that very statement for random ID generation. Usingpuid, you can easily specify "I wantTrandom IDs with a riskRof repeat".puidwill take care of using the correct entropy in efficiently generating the IDs.TOCEfficiencyThe efficiency of generating random IDs has no bearing on the statistical characteristics of the IDs themselves. But who doesn't care about efficiency? Unfortunately, most random string generation, it seems.Entropy sourceAs previously stated, random ID generation is basically atransformationof an entropy source into a characterrepresentationof captured entropy. But the entropy of the source and the entropy of the captured IDis not the same thing.To understand the difference, we'll investigate an example that is, surprisingly, quite common. Consider the following strategy for generating random strings: using a fixed list ofkcharacters, use a random uniform integeri,0 <= i < k, as an index into the list to select a character. Repeat thisntimes, wherenis the length of the desired string. In Python this might look like:fromrandomimportrandintimportstringchars=string.ascii_lowercasen_chars=len(chars)common_id=lambdalen:"".join([chars[randint(0,n_chars)]for_inrange(len)])common_id(8)# => 'wnplkyiz'First, consider the amount of source entropy used in the code above. The Python spec declaresrandom.random()(upon whichrandintdepends) generates 53-bits of precision. So generating an 8 character ID above consumes 8 * 53 = 424 bits of source entropy.Second, consider how much entropy was captured by the ID. Given there are 26 characters, each character represents log2(26) = 4.7 bits of entropy. So each generated ID represents 8 * 4.7 = 37.6 bits of entropy.Hmmm. That means the ratio of ID entropy to source entropy is 37.6 / 424 = 0.09, or a whopping9%. That's not an efficiency most developers would be comfortable with. Granted, this is a particularly egregious example, but most random ID generation suffers such inefficient use of source entropy.Without delving into the specifics (see the code?),puidemploys various means to maximize the use of source entropy. As comparison,puiduses87.5%of source entropy in generating random IDs using lower case alpha characters. For character sets with counts equal a power of 2,puiduses 100% of source entropy.CharactersAs previous noted, the entropy of a random string is equal to the entropy per character times the length of the string. Using this value leads to an easy calculation ofentropy representation efficiency(ere). We can defineereas the ratio of random string entropy to the number of bits required to represent the string. For example, the lower case alphabet has an entropy per character of 4.7, so an ID of length 8 using those characters has 37.6 bits of entropy. Since each lower case character requires 1 byte, this leads to anereof 37.6 / 64 = 0.59, or 59%. Non-ascii characters, of course, occupy more than 1 byte.puiduses the utf-8 character encoding to computeere.The total entropy of a string is the product of the entropy per character times the string lengthonlyif each character in the final string is equally probable. This is always the case forpuid, and is usually the case for other random string generators. There is, however, a notable exception: the version 4 string representation of auuid. As defined inRFC 4122, Section 4.4, a v4uuiduses a total of 32 hex and 4 hyphen characters. Although the hex characters can represent 4 bits of entropy each, 6 bits of the hex representation in auuidare actually fixed, so there is only32*4 - 6 = 122-bits of entropy (not 128). The 4 fixed-position hyphen characters contribute zero entropy. So a 36 characteruuidhas anereof122 / (36*8) = 0.40, or40%. Compare that to, say, the defaultpuidgenerator, which has slightly higher entropy (132 bits) and yet yields anereof 0.75, or75%. Who doesn't love efficiency?TOCOverkill and Under SpecifyOverkillRandom string generation is plagued by overkill and under specified usage. Consider the all too frequent use ofuuids as random strings. The rational is seemingly that the probability of a repeateduuidis low. Yes, it is admittedly low, but is that sufficient reason to use auuidwithout further thought? For example, suppose auuidis used as a key in a data store that will have at most a thousand items. What is the probability of a repeateduuidin this case? It's 1 in a nonillion. That's 10^30, or 1 followed by 30 zeros, or million times the estimated number of stars in the universe. Really? Doesn't that seem a bit overkill? Do really you need that level of assurance? And if so, why stop there? Why not concatenate twouuids and get an even more ridiculous level of "assurance".Or why not be a bit more reasonable and think about the problem for a moment. Suppose you accept a 1 in 10^15 risk of repeat. That's still areallylow risk. Ah, but wait, to do that you can't use auuid, becauseuuidgeneration isn't flexible. The characters are fixed, the representation is fixed, and the bits of entropy are fixed.You could generate the IDs by determining the actual amount of ID entropy required (it's 68.76 bits), selecting some set of characters, calculate the string length necessary given those characters, and finally generate the IDs as outlined in the earlier common ID generation scheme.Whew, maybe that's another reason developers tend to use uuids. That seems like a lot of effort.Ah, but there is another way. You could very easily usepuidto generate such IDs:frompuidimportPuiddb_id=Puid(total=1000,risk=1e15)db_id.generate()'tcDPzTAjoRcU'The resulting IDs have 72 bits of entropy. But guess what? You don't care. What you care is having explicitly stated you expect to have 1000 IDs and your level of repeat risk is 1 in a quadrillion. It's right there in the code. And as added bonus, the IDs are only 12 characters long, not 36. Who doesn't like ease, control and efficiency?Under specifyAnother head-scratcher in schemes that generate random strings is using an API that explicitly declares string length. Why is this troubling? Because that declaration doesn't specify the actual amount of desired randomness, either needed or achieved. Suppose you are tasked with maintaining code that is using random IDs of 15 characters composed of digits and lower alpha characters. Why are the IDs 15 characters long? Without code comments, you have no idea. And without knowing how many IDs are expected, you can't determine the risk of a repeat, i.e., you can't even make a statement about how random the random IDs actually are! Was 15 chosen for a reason, or just because it made the IDs look good?Now, suppose you are tasked to maintain this code:frompuidimportChars,Puidrand_id=Puid(total=500_000,risk=1e12,chars=Chars.ALPHANUM_LOWER)Hmmm. Looks like there are 500,000 IDs expected and the repeat risk is 1 in a trillion. No guessing. The code is explicit. Oh, and by the way, the IDs are 15 characters long. But who cares? It's the ID randomness that matters, not the length.TOCEfficienciesPuidemploys a number of efficiencies for random ID generation:Only the number of bytes necessary to generate the nextpuidare fetched from the entropy sourceEachpuidcharacter is generated by slicing the minimum number of entropy bits possibleAny left-over bits are carried forward and used in generating the nextpuidAll characters are equally probable to maximize captured entropyOnly characters that represent entropy are present in the final IDEasily specifiedtotal/riskensures ID are only as long as actually necessaryTOCtl;drpuidis a simple, flexible and efficient random ID generator:EaseRandom ID generator specified in one line of codeFlexibleFull control over entropy source, ID characters and amount of ID randomnessSecureDefaults to a secure source of entropy and at least 128 bits of ID entropyEfficientMaximum use of system entropyCompactID strings represent maximum entropy for characters usedExplicitClear specification of ID generationfrompuidimportChars,Puidrand_id=Puid(chars=Chars.SAFE32,total=10e6,risk=1e15)rand_id.generate()'RHR3DtnP9B3J748NdR87'TOC