Reworked the solution by replacing clearable widgets with a widget wrapper. This makes it possible to wrap only widgets of nullable fields, and thus not add the clear checkbox when it's not needed.
The implementation is loosely based on RelatedFieldWidgetWrapper from contrib.admin, but uses a bit more Python magic; tested only with TextInput and Textarea widgets and their admin counterparts, but with a bit of luck should work with other widgets too.