Allow modifiers to return the input directly (as in not actually modifying anything)

Previously this would crash as we would have multiple shared pointers managing the
value that didn't know about each other, resulting in a double delete aka a segfault.