Package: custom-literals
custom-literalsA module implementing custom literal suffixes using pure Python.custom-literalsmimics C++'s user-defined literals (UDLs) by defining literal suffixes that can
be accessed as attributes of literal values, such as numeric constants, string
literals and more.(c) RocketRace 2022-present. See LICENSE file for more.ExamplesSee theexamples/directory for more.Function decorator syntax:fromcustom_literalsimportliteralfromdatetimeimporttimedelta@literal(float,int,name="s")defseconds(self):returntimedelta(seconds=self)@literal(float,int,name="m")defminutes(self):returntimedelta(seconds=60*self)print(30.s+0.5.m)# 0:01:00Class decorator syntax:fromcustom_literalsimportliteralsfromdatetimeimporttimedelta@literals(float,int)classDuration:defs(self):returntimedelta(seconds=self)defm(self):returntimedelta(seconds=60*self)print(30.s+0.5.m)# 0:01:00Removing a custom literal:fromcustom_literalsimportliteral,unliteral@literal(str)defu(self):returnself.upper()print("hello".u)# "HELLO"unliteral(str,"u")assertnothasattr("hello","u")Context manager syntax (automatically removes literals afterwards):fromcustom_literalsimportliterallyfromdatetimeimporttimedeltawithliterally(float,int,s=lambdax:timedelta(seconds=x),m=lambdax:timedelta(seconds=60*x)):print(30.s+0.5.m)# 0:01:00FeaturesCurrently, three methods of defining custom literals are supported:
The function decorator syntax@literal, the class decorator syntax@literals, and the
context manager syntaxwith literally. (The latter will automatically unhook the literal
suffixes when the context is exited.) To remove a custom literal, useunliteral.Custom literals are defined for literal values of the following types:TypeExampleNotesint(42).xThe Python parser interprets42.xas a float literal followed by an identifier. To avoid this, use(42).xor42 .xinstead.float3.14.xcomplex1j.xboolTrue.xSinceboolis a subclass ofint,inthooks may influenceboolas well.str"hello".xF-strings (f"{a}".x) are also supported.bytesb"hello".xNoneNone.xEllipsis....xYes, this is valid syntax.tuple(1, 2, 3).xGenerator expressions ((x for x in ...)) are not tuple literals and thus won't be affected by literal suffixes.list[1, 2, 3].xList comprehensions ([x for x in ...]) may not function properly.set{1, 2, 3}.xSet comprehensions ({x for x in ...}) may not function properly.dict{"a": 1, "b": 2}.xDict comprehensions ({x: y for x, y in ...}) may not function properly.In addition, custom literals can be defined to bestrict, that is, only allow the given
literal suffix to be invoked on constant, literal values. This means that the following
code will raise aTypeError:@literal(str,name="u",strict=True)defutf_8(self):returnself.encode("utf-8")my_string="hello"print(my_string.u)# TypeError: the strict custom literal `u` of `str` objects can only be invoked on literal valuesBy default, custom literals arenotstrict. This is because determining whether a suffix was
invoked on a literal value relies on bytecode analysis, which is a feature of the CPython
interpreter, and is not guaranteed to be forwards compatible. It can be enabled by passingstrict=Trueto the@literal,@literalsorliterallyfunctions.CaveatsStabilityThis library relies almost entirely on implementation-specific behavior of the CPython
interpreter. It is not guaranteed to work on all platforms, or on all versions of Python.
It has been tested on common platforms (windows, ubuntu, macos) using python 3.7 through
to 3.10, but while changes that would break the library are quite unlikely, they are not
impossible either.That being said,custom-literalsdoes its absolute best to guarantee maximum
stability of the library, even in light of possible breaking changes in CPython internals.
The code base is well tested. In the future, the library may also expose multiple
different backends for the actual implementation of builtin type patching. As of now,
the only valid backend isforbiddenfruit, which uses theforbiddenfruitlibrary.FeatureStabilityHooking with theforbiddenfruitbackendQuite stable, but may be affected by future releases. Relies on thectypesmodule.Strict mode using thestrict=TruekwargQuite stable, but not forwards compatible. Relies on stack frame analysis and opcode checks to detect non-literal access.Type safetyThe library code, including the public API, is fully typed. Registering and unregistering
hooks is type-safe, and static analysis tools should have nothing to complain about.However, accessing custom literal suffixes is impossible to type-check. This is because
all major static  analysis tools available for python right now (understandably) assume
that builtins types are immutable. That is, the attributes and methods builtin types
cannot be dynamically modified. This goes against the core idea of the library, which
is to patch custom attributes on builtin types.Therefore, if you are using linters, type checkers or other static analysis tools, you
will likely encounter many warnings and errors. If your tool allows it, you should disable
these warnings (ideally on a per-diagnostic, scoped basis) if you want to use this library
without false positives.FAQShould I use this in production?Emphatically, no! But I won't stop you.Nooooooo (runs away from computer)I kind of disagree: yessss (dances in front of computer)Why?Python's operator overloading allows for flexible design of domain-specific languages.
However, Python pales in comparison to C++ in this aspect. In particular, User-Defined
Literals (UDLs) are a powerful feature of C++ missing in Python. This library is designed
to emulate UDLs in Python, with syntactic sugar comparable to C++ in elegance.Butreally, why?Because it's posssible.How? (please keep it short)custom-literalsworks by patching builtin types with custom objects satisfying thedescriptor protocol, similar to
the builtinpropertydecorator. The patching is done trough a "backend", which
is an interface implementing functions to mutate the__dict__of builtin types.
Ifstrict=Truemode is enabled, the descriptor will also traverse stack frames
backwards to the invokation site of the literal suffix, and check the most recently
executed bytecode opcode to ensure that the literal suffix was invoked on a literal value.How? (I love detail)Builtin types in CPython are implemented in C, and include checks to prevent
mutation at runtime. This is why the following lines will each raise aTypeError:int.x=42# TypeError: cannot set 'x' attribute of immutable type 'int'setattr(int,"x",42)# TypeError: cannot set 'x' attribute of immutable type 'int'int.__dict__["x"]=42# TypeError: 'mappingproxy' object does not support item assignmentHowever, these checks can be subverted in a number of ways. One method is to use
CPython's APIs directly to bypass the checks. For the sake of stability,custom-literalscalls thecurse()andreverse()functions of theforbiddenfruitlibrary
to implement these bypasses. Internally,forbiddenfruituses thectypesmodule
to access the C API and use thectypes.pythonapi.PyType_Modified()function to
signal that a builtin type has been modified. Other backends may also be available in the future,
but are not implemented at the moment. (As an example, there is currently a bug
in CPython that allowsmappingproxyobjects to be mutated without usingctypes.
This was deemed too fragile to be included in the library.)Python's@propertydecorator
implements thedescriptor protocol.
This is a protocol that allows for custom code to be executed when accessing specific
attributes of a type. For example, the following code will print42:classMyClass:@propertydefx(self):print(42)MyClass().xcustom-literalspatches builtin types with objects implementing the same protocol,
allowing for user-defined & library-defiend code to be executed when invoking a literal
suffix on a builtin type. It cannot however use@propertydirectly, as elaborated
below.The descriptor protocol is very flexible, used as the the backbone of bound methods,
class methods, and static methods and more. It is defined by the presence of one
of the following methods*:classSomeDescriptor:# <instance>.<attribute>def__get__(self,instance,owner)->value:...# <instance>.<attribute> = <value>def__set__(self,instance,value)->None:...# del <instance>.<attribute>def__delete__(self,instance)->None:...*and optionally__set_name__The descriptor methods can be invoked from an instance (some_instance.x) or from
a class (SomeClass.x). Importantly for us, the__get__method is called with
different arguments depending on whether the descriptor is accessed from an instance
or a class:classMyDesciptor:def__get__(self,instance,owner)->value:print(f"Instance:{instance}")print(f"Owner:{owner}")classMyClass:x=MyDesciptor()MyClass().x# Instance: <__main__.MyClass object at 0x110e3a170># Owner: <class '__main__.MyClass'>MyClass.x# Instance: None# Owner: <class '__main__.MyClass'>This is used to differentiate between the two cases.@property's implementation
simply returns the descriptor instance ifinstanceisNone, which is a fair
test for whether the descriptor is accessed from a class or an instance.Keen-eyed readers may notice however that this is not a perfect test. What ifMyClassis somehowtype(None)? In this case, the two cases will be indistinguishable.
In normal code, this is not a problem, astype(None)is a builtin type, and
thus cannot be mutated. Incustom-literals, however, this breaks custom literals
that are defined ontype(None).This can thankfully be mitigated thanks to the concept of adata descriptor.
A data descriptor is a descriptor that defines__set__or__delete__. When
Python tries to resolve attribute access on an instance, it will first check whether
itstypehas a data descriptor for the attribute, overriding any descriptors
defined on theinstanceitself. For example, suppose the following example using
a metaclass (a class inheriting fromtype):classDataDescriptor:def__get__(self,instance,owner):print("The data descriptor was called!")print(f"Instance:{instance}")# Simply the presence of the method is enough# to convert this into a data descriptordef__set__(self,instance,value):raiseAttributeErrorclassNormalDescriptor:def__get__(self,instance,owner):print("The normal descriptor was called!")print(f"Instance:{instance}")classMyMeta(type):x=DataDescriptor()classMyClass(metaclass=MyMeta):x=NormalDescriptor()MyClass.x# The data descriptor was called!# Instance: <class '__main__.MyClass'>MyClass().x# The normal descriptor was called!# Instance: <__main__.MyClass object at 0x10f468ee0>sThis example shows that it is possible to ensure that a descriptor is always
called on an instance withinstanceset to an instance of the class. In the case ofcustom-literals, this is achieved by patching a data descriptor (any data descriptor)
ontypewhentype(None)is also being patched. This removes the ambiguity of
whether the descriptor is called on an instance or a class. Yay!Finally,custom-literalsalso provides a mechanism for optionally detecting when a custom
literal suffix is invoked on a constant and literal type. (This is invoked when thestrictargument is set toTrue.) This is achieved by attaching
extra code to the__get__method of the custom literal descriptor. The code performsbytecode analysisat the invocation site of the custom literal suffix.The CPython interpreter uses stack frames to implement function calls. When a function is
called, a new frame is created and pushed to the stack, and when the function returns, the
frame is popped off the stack. Importantly, these frame objects can be accessed directly
from Python:importinspectdeffoo():local_variable=123bar()defbar():# Alternatively, use `sys._getframe()`frame=inspect.currentframe()# The `f_back` attribute of a frame object# points to the frame that called itprevious_frame=frame.f_back# Frame objects have information about the# invocation context of the frame, including# e.g. local variablesprevious_locals=previous_frame.f_localsprint(previous_locals['local_variable'])# 123Thef_codeattribute of a frame object contains information about the bytecode of the
currently executed code. CPython being an interpreter, this bytecode corresponds roughly
to the source code of the function. For example, see the disassembly of the following:importdisdefadd(a,b):c=a+breturnint(c)dis.dis(add)# Outputs:# 4           0 LOAD_FAST                0 (a)#             2 LOAD_FAST                1 (b)#             4 BINARY_ADD#             6 STORE_FAST               2 (c)## 5           8 LOAD_GLOBAL              0 (int)#            10 LOAD_FAST                2 (c)#            12 CALL_FUNCTION            1#            14 RETURN_VALUEFirst, the two argumentsaandbare pushed onto the stack.The arguments are popped from the stack and used as the operands for+. The result is pushed onto the stack.The top of the stack is popped and stored in a local variablec.Theintfunction is fetched from the global namespace and pushed to the stack.The local variablecis pushed to the stack.Theintfunction is called with one argument, and the return value ofintis pushed to the stack.The result is popped from the stack and returned.In the case of custom literals, the opcodes we are concerned about are the following:LOAD_CONST, used to load a constant (including most literal values) to the stackBUILD_TUPLE/BUILD_LIST/BUILD_SET/BUILD_MAP, used to push tuple/list/set/dict literals to the stackFORMAT_VALUE, used to push a formatted f-string literal (f"{a} {b} {c}") to the stackLIST_TO_TUPLE/LIST_EXTEND/SET_UPDATE/DICT_UPDATE, sometimes used in list/set/dict literals, for example when using the star unpack syntax ([a, b, c, *x])(Do keep in mind that opcodes are not necessarily forwards compatible. Python 3.11 could release
a dozen new opcodes tomorrow that need to be accounted for by the library! This is whycustom-literalsdoes not perform bytecode analysis by default.)If strict mode is enabled, the library will traverse up through the stack frames, inspect the bytecode,
check the most recently executed opcode (available inframe.f_lasti), and check if it is one of the
opcodes listed above. If the opcode is not in the allowed list, an error is raised, which is why
the following code raises an error:@literal(str,strict=True)defxyz(self):return123abc="abc"abc.xyz# TypeError: the strict custom literal `xyz` of `str` objects can only be invoked on literal valuesPutting all of these features together,custom-literalsis able to do the seemingly impossible -
define custom literal suffixes on builtin types that can only be invoked on literal values!Making this project has been a fascinating deep dive into some of the internals of CPython, and
I hope it has been equally interesting to you, the reader.Could this ever be type safe?I doubt it. The assumptions made by static analysis tools are incredibly useful, and
this is such an edge case it makes no sense for them to assume builtin literal types can have
dynamically set attributes. In addition, there isn't a good way to signal to your type
checker that an immutable type is going to be endowed with new attributes!License(c) RocketRace 2022-present. This library is under the Mozilla Public License 2.0.
See theLICENSEfile for more details.ContributingPatches, bug reports, feature requests and pull requests  are welcome.LinksGitHub repositoryPyPI