# Synthesizing Standalone World-Models, Part 3: Dataset-Assembly

*This is part of a series covering* [*my current research agenda*](https://www.lesswrong.com/posts/LngR93YwiEpJ3kiWh/research-agenda-synthesizing-standalone-world-models)*. Refer to the linked post for additional context.*

This is going to be a very short part. As I'd mentioned in the initial post, I've not yet done much work on this subproblem.  
 

* * *

From [Part 1](https://www.lesswrong.com/posts/xvCNjLL3GZ6w2BWeb/synthesizing-standalone-world-models-part-1-abstraction), we more or less know how to learn the abstractions given the set of variables over which they're defined. We know their type signature and the hierarchical structure they assemble into, so we can just cast it as a machine-learning problem (assuming a number of practical issues is solved). For clarity, let's dub this problem "**abstraction-learning**".

From [Part 2](https://www.lesswrong.com/posts/kNyMwXQxctWtaRZhs/synthesizing-standalone-world-models-part-2-shifting), we more or less know how to deal with shifting/resampled structures. While the presence of specific abstractions doesn't uniquely lock down what other abstractions are present at higher/lower/sideways levels, we can infer a probability distribution over what abstractions are likely to be there, and then resample from it until finding one that works. Let's call this "**truesight**".

Except, uh. Part 1 only works given the solution to Part 2's problem, and Part 2 only works given the solution to Part 1's problem. We can't learn abstractions before we've stabilized the structure/attained truesight, but we can't attain truesight until we learn what abstractions we're looking for. We need to, somehow, figure out how to learn them jointly.

This represents the third class of problems we need to solve: figuring out how to transform whatever data we happen to have into datasets for learning new abstractions. Such datasets would need to be isomorphic to samples from the same fixed (at least at a given high level) structure. Assembling them might require:

*   Figuring out what variables to consider.
*   Figuring out what *samples* of those variables to consider. (E. g., when learning Conway's glider, we'd only consider the samples of the first $[0:3]\times [0:3]$ set of cells for the first few time-steps. Similarly, in fluid dynamics, we use [material derivatives](https://en.wikipedia.org/wiki/Material_derivative) to lock our viewpoint to a specific particle in a stream, instead of to a specific point in space – switching the representation from "spatial points are the low-level random variables" to "particles are the low-level random variables".)
*   Figuring out what functions of those variables to consider. (Which might involve effectively discarding specific "subvariables" $x_i^j$ of a given variable $x_i$, or treating a bunch of variables as subvariables to get at their synergistic information.)

Call this "**dataset-assembly**".

Dataset-assembly has some overlap with the truesight problem, so the heuristical machinery for implementing them would be partly shared. In both cases, we're looking for functions over samples of some known variables that effectively sample from the same stable structure. The difference is whether we already known that structure or not.

Another overlap is with the heuristics I'd mentioned in [1.6](https://www.lesswrong.com/posts/xvCNjLL3GZ6w2BWeb/synthesizing-standalone-world-models-part-1-abstraction#1_6__Summary), the ones for figuring out which subsets of variables to try learning synergistic/redundant-information variables for (instead of doing it for all subsets). Indeed, given the shifting-structures problem, those are actually *folded into* the heuristics for assembling abstraction-learning datasets!

Introspectively, in humans, "dataset-assembly" is represented by qualitative research as well, and by philosophical reasoning (or at least [my model](https://www.lesswrong.com/posts/LbfWAq4F8tHaQsTyT/towards-the-operationalization-of-philosophy-and-wisdom) of what "philosophical reasoning" is). "Dataset-assembly heuristics" correspond to research taste, to figuring out what features of some new environment/domain to pay attention to, and which parts of reality could be meaningfully grouped together and decomposed into a new abstract hierarchy/separate field of study.

My thinking on the topic of dataset-assembly is relatively new, and isn't yet refined into a proper model/distilled into something ready for public consumption. Hence, this post is little more than a stub.

That said, I hope the overall picture of the challenge is now clarified. We need to figure out how to set up a process that jointly learns the heuristics for solving these three classes of problems.

* * *

What I'm particularly interested in here for the purposes of [the bounties](https://www.lesswrong.com/posts/LngR93YwiEpJ3kiWh/research-agenda-synthesizing-standalone-world-models#Bounties) is... well, pretty much anything related, since the map is pretty blank. Three core questions:

*   What are high-level ways to formalize the dataset-assembly subproblem?
*   What are some heuristics for solving this subproblem?
*   How should we think about/model the problem of solving all three subproblems jointly?

(My go-to approach in such cases is to figure several practical heuristics, go through a few concrete cases, then attempt to distill general principles/algorithms based on those analyses.)