Recently we saw advent of multimodal transformers audio-vision-language models, aka AVLM. But their utility is limited, because they are mostly speech/language bound -- describing visual content or transcribing audio. Not all aspects can be adequately represented in linguistics form; and sometimes we need estimate "experience" of perception. 
What if we were to combine AVLM with data analysis algorithms, "psychoacoustics model" and "visual psychophysics model" to provide "approximation of human experience", how people might have percived content? Actually, in some cases we would prefer to represent input as a program, which means additional modules for "inverse graphics" or "neuro-symbolic program induction", which would allow precise manipulations through code snippet like commands/actions. So, there would be significant difference between "barebone AVLM" and proposed "hybrid AVLM with proxy models". 
From pragmatic perspective, we want this to serve as a means to improve quality -- difference between ASR that would simply predict most likely words and occasionally mess important part vs "hybrid AVLM" that would notice that "something went wrong, maybe try another approach or add disclaimer"; or ask to generate music and get mediocre results vs system which would iterate on its own until it did satisfactory piece of music. Of course it would involve capacity to decompose task, chain actions and control results -- aspects that are missed in current generation of LLMs/AVLMs (in other words, in cases where human person would see opportunity and make intermediate steps, this models would ignore situation and would wait explicit instructions and pray they would not "forget important details", which quickly become exhausting). 
Not sure how to codify "there is many strategies to work with data" wisdom. Part of valuable signal comes from inside system itself, so we might need to collect and utilize it as well, let's call it "Self-Referential Logging Mechanism". 
Perhaps we need to introduce RL agent to serve as a "high level control module". The difference is not NMT for language translation vs language model which can do some other tasks as a bonus; more like "monolithic reactive" model vs hybrid modular architecture with agentic capabilities (RL control module); as a bonus, it should be possible to work asynchronously if we would organise "modus operandi" around "events queue" where user input (messages, requests, files attached, etc) are represented as such events; asynchronous processing extends to internal "action chains" and abstraction of output generation as a special action, so input and output can be processed simultaneously with "interruption handling or mid response adjustments" strategies if we move towards even "smarter" agency; it might lead to situation where AVLM role is gradually reduced to "interface" (input "parser" and user facing output "keyboard"/"communicator") receiving data for processing (filtered and "compiled" context with only relevant segments) according to instructions, both provided by Control module. I'm uncertain how to make Control (short name for RL agent component) work with "soft" action space where it is only partially defined, and can be "populated from solutions search, reasoning and reflection on experience"; the "goal" at higher levels of cognitive sophistication definition is also somewhat arbitrary (which is true for both humans and advanced machine learning architectures as we see here), possibly involving context depending balancing of pre-defined objectives with inverse reinforcement learning and informed by "anticipatory framework". Again, important parts is capabilities improvement and robustness (to see failures before user would point, to do not ignore signals that "maybe it was bad/dumb move"), not "emulation of human perspective/perception for entertainment". 
What am I talking about? This is getting complex, identify key themes and technologies mentioned. If there are uncertainty points, highlight them with clarification questions.