Patent AU-2023203193-A1 (AU)
Title: Methods and systems for autonomous task composition of vision pipelines using an algorithm selection framework
Inventor: ROY CHOUDHURY ABHISHEK
IPC: G06V10/94
Filed: 2023-05-22
Abstract: This disclosure relates generally to systems and methods for autonomous task 
composition of vision pipelines using an algorithm selection framework. The 
framework leverages transformer architecture along with deep reinforcement 
learning techniques to search an algorithmic space for unseen solution templates. In 
an embodiment, the present disclosure describes a two stage process of identifying 
the vision pipeline for a particular task. At first stage, a high-level sequence of the 
vision pipeline is provided by a symbolic planner to create the vision workflow. At 
second stage, suitable algorithms for each high-level task are selected. This is 
achieved by performing a graph search using a transformer architecture over an 
algorithmic space on each component of generated workflow. In order to make the 
system more robust, weights of embedding, key and query networks of a visual 
transformer are updated with a Deep Reinforcement Learning framework that uses 
Proximal Policy Optimization (PPO) as underlying algorithm.  
[To be published with FIG. 4]
4/9 
Receiving, (i) a plurality of input data pertaining to one or more domains of one or more 
enterprises and (ii) a descriptive query from a user as input, wherein the plurality of input data 202 
comprises one or more input parameters, one or more domain requirements and corresponding 
solutions, and wherein the descriptive query describes a goal task to be executed on the 
plurality of input data 
Identifying, a vision pipeline for execution of the goal task by inputting the descriptive query 
and one or more state attribute levels corresponding to the plurality of input data to a symbolic 204 
planner, wherein the symbolic planner dynamically composes one or more subtasks associated 
with the goal task and constructs a Directed Acyclic Graph (DAG) for each of the one or more 
subtasks using a parser parser 
Identifying, a set of algorithms from a plurality of algorithms that are suitable to be executed 
at one or more stages of the vision pipeline for execution of the goal task using a transformers 
and Reinforcement Learning (RL) based autonomous pipeline composition framework, 
wherein the transformers and Reinforcement Learning based autonomous pipeline 
composition framework comprises a set of RL policies that are interlinked and resemble every 
step in the vision pipeline, and wherein each RL policy comprises: (i) a task specific module 
comprising the plurality of algorithms that performs a specific sub task from the one or more 206 
subtasks associated with the goal task; (ii) an embedding module comprising one or more 
neural networks corresponding to each algorithm in the plurality of algorithms comprised in 
the task specific module, wherein each fully connected neural network of the embedded 
module is configured to map output of each algorithm in the subset of algorithms to a specific 
embedding output dimensionality; and (iii) a transformer module comprising a key network 
and a query network, wherein the key network converts an embedding output of each of the set 
of algorithms into a key vector and the query network receives an aggregation output of the 
embedding output of each of the subset of algorithms to generate a global query vector 
Dynamically Configuring, the vision pipeline for execution of one or more goal tasks in one or 
more environment and system configurations 
200 
FIG. 4