The Data Æther
By: George Svarovsky
Tags: Crdt, Programming Languages, Data, Coding, Rdf

We’re Not Done

There are some niggles that make us wonder how X can exist in the real world. One is that “native” data structures aren’t consistent. I might love Haskell or C# or Python or Prolog (I might!). While I can probably express most of X in my language, my mileage is going to vary. A class in Java is not the same as a class in Javascript. (Can I add a property?) Even basic lists and maps and sets are not the same. (Equality? What about null and undefined and none and empty ?)

Languages do not express data structures in the same or compatible ways.

When it comes to data semantics, at least most programming languages overwhelmingly have one thing in common: they suck at it. All the big, popular languages are imperative. That means they are great at saying ‘do this, do that’, but they have real trouble saying ‘this makes sense’. If I look at a class in Java, it’s really hard to see all the things that are always true, and the things that need to be true before some change is allowed, and the things that are true after those changes. If I’m lucky, someone has added some comments, or assertions, or written some test cases. (Or interface specifications, if I’m really lucky.)

But these things are in the code, and that means they’re re-coded everywhere the data appears. Or not, in which case the bug takes a little while to manifest when the data arrives at somewhere that notices (if it ever does). So, how do I know that a pre-condition was checked when all I get is the post state?

Custom semantics can be hard to express: invariants, allowed operations, inferences. And they must be re-expressed wherever the data appears, or risk later failures.

We want this stuff underneath abstraction X, so my code is guaranteed not to break the rules of the data (preferably at compile-time) but any time is better than never.

When it comes to truth, things are even more complicated. Languages come with baggage. Mutability. Threading. “Volatile” is a thing. This is not going well. Looks like we have a choice. On the one hand, we could reinvent programming; stop using our favorite languages, and move everyone to a language that has abstraction X built-in. But that’s a concept for another article, I think.

So let’s look at the other hand, where we back off from this idea of everything working completely natively. After all, everyone already has to deal with mapping one abstraction onto another, all the time. Let’s just assert what we want about X, and let clever programmers bind X to their language with libraries.

Requirements for Binding X to Code

Once X is bound to a language with a library, we’ll have a syntax for that language. I’d say it’s table stakes that anything that can be expressed natively is; but really that’s up to the binding library author (as they are the expert on that language).

There’s one quality that stands out, because it’s not well-supported by languages in general, and that’s having a universal address space. If we’re truly going to abstract away the vagaries of media and protocols, then we need a way to refer to some data not as a file on a filesystem, or a row in a database, or a memory location.

Beyond that, we need to be able to express and enforce the data structure and rules in X so that the code we write doesn’t break the data. Again, it would be nice if we could do this at compile-time, but it’s also important at runtime. Why? Because most applications have at least some variability in their data on a per-install, per-customer, or even per-session basis. As app developers, we often try to tuck this into carefully managed corners as ‘customization’ features. I have spent a lot of time in that corner.

Data variability at runtime should be a first-class concept.

Truth is the axis for which we don’t have prior art. I haven’t come across any language that addresses it consistently and head-on. We haven’t really got a canonical way to say this data is this far away, and this much out of date, and these will be the consequences of you editing it. There are some ideas — for example, you should check out the proposed Braid HTTP extension.

And naturally, for all of this, if things are going to change — and we know they will — we need a well-defined way to cope with that.

The Consequences of X

So let’s say we’ve done it, we have X and it’s available to our choice of programming language. What would we expect to be different?

The most important thing is that app developers will spend less time wrangling data through protocols and formats and layers. A programmer only needs to learn the correspondence between their platform and X.

But engineers who like doing those things can still get their kicks by developing more and better ways for X to get from A to B, knowing that their efforts will be recognized because the data itself advertises the result.

Physical protocols will only be relevant to those choosing or optimizing them.

And let’s notice one more thing. With X, data is not flowing anymore. I don’t have to think in terms of requesting and receiving and decoding and validating and locking and (briefly) adding value, and then encoding and responding and emitting and unlocking, and am-I-done-yet?

Instead, data is just there, bathing my code, and just a de-reference of an identifier away. Sometimes it’ll take a few milliseconds to arrive, but I already know how long and I can build those expectations into my software design. And tomorrow, when someone clever has noticed that I keep asking for such-and-such data, and makes it available to me before I ask, I can progressively enhance that design.

Final Thoughts

As the author of m-ld, I have my own ideas about how to achieve X, and the Data Æther. I’ll go into those next time. But what do you think? Can we get away from the endlessly expensive but worthless wrangling of syntax, semantics, and truth?

And what would we build if we could?

Update: Let me tell you about where this vision has led me, personally. I’m a practical type, and I’m only comfortable selling hopes and dreams if I can show that important parts of the dream work in a real life…