Patent Description:
Methods of artificial intelligence are applied to at least partially autonomous devices to enable supervised or unsupervised autonomous operations. To ensure that the apparatus operates within a given legislative framework and within given general conditions, test cases are applied to the device. In a test case, a reaction of the apparatus to a test situation is compared to a desired reaction to determine a result of the test.

<NPL> discloses aspects of such methods.

With currently available testing approaches of brute-force it is both time consuming and resource consuming to ensure that the apparatus operates at all times as expected. It is also difficult to provide transparent record of proof of the safe operation. It is therefore desirable to provide an improved testing scheme.

This is achieved by the method and the device according to the independent claims.

In the description, the term testing refers to validation, certification or homologation of an apparatus or to testing and certification of an operator thereof. Testing in this context may also refer to testing a dynamic system. The testing is applicable in the automotive field as well as in the field of aviation or industrial automation.

The apparatus is an at least partially autonomous vehicle or a configurable component thereof. The apparatus may be an aircraft or a robot, in particular a mobile robot. The apparatus may be an at least partially autonomous mobile or stationary robot, an at least partially autonomous land vehicle, an at least partially autonomous air borne vehicle, e.g. a drone, or an at least partially autonomous watercraft.

The term principle in the description refers to a principle defined by a legal framework or a general condition. An example for a principle for an at least partially autonomous vehicle may demand exercise maximum diligence when a child is at the side of a road because the child could step onto the lane unexpectedly.

The term action in the description refers to an action that is presented to the apparatus or a user thereof to prompt for a reaction. In a test environment for an operator, the apparatus may prompt the operator via a human machine interface to perform the reaction. A reaction by the operator may directly influence the behavior of the apparatus or may answer a question posed by a test case. For the at least partially autonomous vehicle, the reaction may be defined by a rule indicating to reduce the speed of the vehicle when a child is detected at a side of a road. The term context in the description refers to aspects of an environment and/or a situation in which the apparatus operates. An example for a situation the at least partially autonomous vehicle operates in may be defined by an image depicting a child standing at a side of a road in the context. Likewise, a video sequence showing the child moving at the side of the road in the context may define a situation. The environment may define a weather condition, e.g. summer, winter, twilight, night, rainfall, snowfall.

The term test case refers to an action and the context for the action. The test case can be executed in a simulation environment or in real world environment.

The term scenario in the description refers to a plurality of contexts or contexts that may be combined with an action to form a concrete situation for the scenario. An exemplary scenario defines situations that are defined by the image depicting the child standing at a side of the road in various weather or illumination conditions including but not limited to rain, snow, sunshine, in twilight, at night, at backlight. Likewise, a video sequence showing the child moving at the side of the road in these conditions may define the situation.

The test case according to the following aspects is defined for testing if a principle is met. More specifically, a test case suitable for testing the principle is determined.

A device for testing is provided in accordance with claim <NUM>. Thus, rules for testing can be extracted by the device for testing for testing whether a principle is met depending on scenarios.

The learning arrangement may be adapted to determine a similarity between two scenarios and the modelling arrangement may be adapted to determine, store and/or output the model comprising only one of the two scenarios, when a similarity between the two scenarios is detected and to determine, store and/or output the model comprising both otherwise. Thus, two similar scenarios are not tested twice.

The learning arrangement may be adapted to prompt for user input to determine a similarity between two scenarios and the modelling arrangement may be adapted to determine, store and/or output the model comprising only one of the two scenarios when a similarity between the two scenarios is detected from user input and to determine, store and/or output the model comprising both otherwise. Thus, the similarity may be detected by a human operator interaction.

The learning arrangement may be adapted to determine a linear independent Eigenspace of scenario vectors representing the scenarios, wherein each of the scenario vectors defines a rule for combining at least one action with at least one context. The linear independent Eigenspace of scenario vectors comprises the rules that map actions and contexts to scenarios unequivocally.

The modelling arrangement may be adapted to determine a test case defined by the model depending on the rule. The rule maps the actions and contexts to test cases.

The test case may be defined depending on at least one control signal for the apparatus or a component thereof, or depending on at least one sensor signal, in particular an audio signal, a video signal, a radar sensor signal, an ultrasonic sensor signal or a time of flight sensor signal. Thus, the device may test the respective sensors or a behavior of the apparatus directly.

A corresponding method is provided in accordance with claim <NUM>. Thus, the rule is learned in a training, e.g. from labeled or unlabeled data representing the scenarios and principles. The labels may indicate which scenarios suitable for testing which of the principles. The rules may be determined to match suitable scenarios to the respective principle.

The method may comprise determining a similarity between two scenarios and determining, storing and/or outputting the model comprising only one of the two scenarios, when a similarity between the two scenarios is detected and determining, storing and/or outputting the model comprising both otherwise. This provides a very efficient device for testing.

The method may comprise prompting for user input to determine a similarity between two scenarios and determining, storing and/or outputting the model comprising only one of the two scenarios when a similarity between the two scenarios is detected from user input and determining, storing and/or outputting the model comprising both otherwise. This allows a selection in the process of training the device.

The method may comprise determining a linear independent Eigenspace of scenario vectors representing the scenarios, wherein each of the scenario vectors defines a rule for combining at least one action with at least one context. The method may comprise determining a test case defined by the model depending on the rule.

Preferably, the test case is defined depending on at least one control signal for the at least partially autonomous apparatus or a component thereof, or depending on at least one sensor signal, in particular an audio signal, a video signal, a radar sensor signal, an ultrasonic sensor signal or a time of flight sensor signal.

Further advantageous aspects are derivable from the following description and the drawings. In the drawings.

The term model refers to a model that is defined by expert knowledge or has been determined using self-learning, e.g. from labelled data. The model as described in the following contains a mapping of test cases to principles that shall be tested. The model may be a probabilistic network or computational intelligence. The model may comprise a Bayesian network. Fuzzy logic type I or type II or a machine learning system may be used to implement the model as well. Mapping the Bayesian network to a multi-layer feedforward neural network, conditional probabilities of the Bayesian network may be learned by the Bayesian network from data. A probability distribution may be determined from the Bayesian network model by an analysis of the structure of the network. The model may be based on fuzzy logic (type I or type II) or model dependencies of test cases using Petri nets or similar state based modelling approaches. With such dependencies, experts may define a transparent mapping in the model by rules and assumed or generated probabilities.

Bayesian networks are an example for implementing the model, in particular to model conditional probabilities in a probability distribution. Other methods such as deep neural networks may be used.

The term rule refers to machine-readable rules. Decision trees may represent these rules, by formal logic or by fuzzy logic (type I or type II).

The term general condition refers to conditions that apply to the rules. The general conditions may be implemented in machine-readable form, e.g. by MySQL or SPARQL.

<FIG> depicts a device <NUM> for testing an apparatus. The device is applicable for testing of a component of the apparatus or an operator of the apparatus as well. In the device, actions a1,. , an and contexts K1,. , Km define test cases. A test case * with actions a1, a2,. an in context Km is depicted schematically in <FIG>. For the test case * a probability σIII is depicted that indicates the relevance of the test case * for testing a principle PIII. In <FIG> a plurality of principles PI,. , Pn are schematically depicted. Also depicted in <FIG> is another probability σz indicating the relevance of another test case in the context K1 for the principle Pn.

In an exemplary implementation, rules Ri firm up the principles PI,. In an example i = <NUM> rules may be used. The mapping of principles PI,. , Pn to rules R1,. , Ri may be implemented in a lookup table where x indicates a Rule that firms up a certain principle:.

To test that a principle PI,. , Pn is met, a plurality of the actions a1,. , an are applied in a plurality of contexts K1,. To determine to which extend a certain principle is met, the correlation of actions, contexts and rules is evaluated. The correlation is represented by probabilities σI,. , σz linking actions, contexts and probabilities to principles.

The actions and contexts are mapped to the probabilities in the model. The model is depicted in <FIG> with reference sign <NUM>. A Bayesian network {aj, σj, kj} may represent the model, where j is an index for a link to a particular principle Pj. The Bayesian network may be trained based on test cases e.g. by a training method for artificial neural networks. Test cases for the training are for example determined depending on a rule-based model <NUM>, for instance a fuzzy set model (type I or type II), and/or depending on a relational database <NUM> or a case-based reasoning database <NUM>. The test cases may be stored in the case-based reasoning database <NUM> or a relational database <NUM> as well. The test cases * are for example stored with and accessible by an index.

The mapping of contexts K1,. , Km and actions a1,. , an to test cases * may be implemented in a lookup table:.

By way of example, a rule Rj concretizes a principle Pj as follows:.

Preferably, the rational database <NUM> and the rule-based model <NUM> are a predefined set of fundament principals manually created based on legal and ethical aspects. They form a basis against which the apparatus <NUM> or the user thereof has to be tested. The rational database <NUM> and the rule-based model <NUM> may be merged to a to a relational database <NUM> which contains principles Pj and corresponding action rules Rj.

The case-based reasoning data-base <NUM> in this example contains an amount x of video data Rjx for different contexts Kx of the contexts K1,. The contexts are e.g. video data of different children, a group of children, holding the hand of an adult, with a bicycle, in summer, in snow, in dim light.

The relational database <NUM> in this example contains labelled test cases, e.g. the video data with the corresponding label. A human may label the video data appropriately.

In this example, the conditional probability σj provides a statement for the corresponding actions aj and context Kj. In other words, the conditional probability σj provides the probability for violating the principle Pj in different context Kj of the contexts K1,. The context Kj may for example define "summer in Arizona", "winter in the Black Forrest".

In the example, the principle Pj is the index for the corresponding rules from the table above. In other words, the rules Ri for the principle Pj are determined from the table. In addition, the test case, i.e. the action ai and the context Ki, are determined depending on the rule Ri.

For complex scenarios, a plurality of actions and contexts may be required to cover the principles Pi. The rules Ri with a high probability σi may be used to define the complex scenarios. For example the actions "A Child standing at the side of the road" is combined with one or more of the contexts "in winter", "in summer", "in twilight", "at night", "in the rain" or a combination thereof.

The test cases * are applied for example at an at least partially autonomous apparatus <NUM> for testing. The device <NUM> for testing at the at least partially autonomous apparatus <NUM> is depicted in <FIG>.

The device <NUM> comprises a learning arrangement <NUM>. The device <NUM> comprises a modelling arrangement <NUM>, a selecting arrangement <NUM> and a testing arrangement <NUM>.

The learning arrangement <NUM> may be adapted to determine at least one rule from scenarios and principles and/or reactions observed for the apparatus <NUM> in response to a test case. Each of the rules maps the actions a1,. , an and contexts K1,. , Km to a test case. Different previously unknown actions or contexts may be implemented. The scenarios and/or principles may be labeled data provided to the learning arrangement <NUM>.

The learning arrangement <NUM> may be adapted to determine the at least one rule depending on the rule-based model <NUM>, e.g. the fuzzy set model (type I or type II).

<FIG> depicts aspects of the creation of the model <NUM>. <FIG> depicts aspects of the creation of elements of the case-based reasoning database <NUM>. <FIG> depicts aspect of approaches for creating the model <NUM> by automatic learning.

The modelling arrangement <NUM> is adapted to determine the model <NUM> depending on the at least one rule.

The relational database <NUM> contains principles Pj and is linked to the corresponding action rule Rj in the rule-based model <NUM>. For those, rules and principles have to be defined in the model <NUM>.

The task of the modelling arrangement <NUM> is to link those fundamental rules with action a1,. , an, context K1,. , Km and likelihoods σ1,. , σz which are related to executable test cases, e.g. video sequences, Radar or Lidar data sequences in the case-based reasoning data-base <NUM> and/or the relational data-base <NUM>.

For this purpose, as depicted in <FIG>, a human operator 20a may compile the model <NUM> and link the relations to the principles in the relational database <NUM> and the rules in the rule-based model <NUM> as well as test cases in case-based reasoning database <NUM> and/or the relational database <NUM>.

The data structure of the model <NUM> is for example a set of links which can be in the form a matrix, a Rule-tree, e.g. according to fuzzy logic type II, or a net linking the multiple elements of the case-based reasoning data-base <NUM> and/or the relational data-base <NUM> with the principles.

This is a "white box" approach as the human operator has an overview and connects the test cases and principals manually due to his knowledge. The human operator may be assisted by a computer-aided system, which helps to establish and administrate the links.

In an example, a human operator 20a interacts with the modelling arrangement <NUM> to determine the model <NUM> based on knowhow about all relationships and sets the links and likelihoods.

Once the generated structure of the model <NUM> is decided, the case-based reasoning database <NUM> can be filled with multiple models of actions and with different contexts K1,. , Km that can serve as test scenarios automatically.

For instance, one particular scenario comprising a playing child in different appearances such as "in sunset", "in rain", "in snow" can be automatically created.

Automatic learning by the learning arrangement <NUM> is described below with reference to <FIG>. The learning arrangement <NUM> is in one example adapted to automatically create a Bayesian net based on database structure of the relational database <NUM> as well as the relational database <NUM> and the rule-based model <NUM>.

The human operator 20a may structure the databases in actions and contexts and may suggest relations to principles and rules. In an exemplary machine learning approach, the human operator 20a is providing the relation database <NUM> with labeled scenarios. For instance, the relation database <NUM> can be manually pre-structured according to principles and an action description.

, a video sequence for an action a1 is assigned "a child playing on the road"; a video sequence for an action a2 is assigned "a child behind a car". Other actions may be assigned alike.

When labels are used in the database scenarios, these relate to scenarios and principles of the rational database <NUM> and the rule-based model <NUM>.

The learning arrangement <NUM> deduces automatically, e.g. by an optimization algorithm, the concise relationships σI,. , σz for all PI,. , Pn and a1,. , an of K1,. , Km in terms of an automatically created e.g. trained Bayesian network for all training data e.g. all labeled scenario and all labeled principles. The learning arrangement <NUM> may train the Bayesian network based on labeled test cases contexts and/or actions as well. The learning arrangement <NUM> may reject concise or fuzzy test cases.

This is a "Black box" approach as the human operator 20a is only doing generic labeling and the learning arrangement <NUM> is deducing the relationship automatically.

The model <NUM> in the example correlates actions a1,. , an in contexts K1,. , Km with the probability distribution σ1,. , σz indicating the relevance of the actions a1,. , an in the contexts K1,. , Km regarding test cases for testing whether the apparatus <NUM> operates according to a principle PI,. , Pn or not.

The model <NUM> in the example defines an individual probability for an action in a context. In the example, the model <NUM> defines individual probabilities σ1,. , σz for the same action in different contexts.

The model <NUM> is in one aspect created and implemented by indexing between PI,. , Km and σ1,.

The model <NUM> relates to test cases in the case-based reasoning database <NUM> and the relational database <NUM>, e.g. to video and other test data stored therein. The case-based reasoning database <NUM> and the relational database <NUM> are examples for databases, which contain executable test sequences in form of videos, Lidar, or Radar data sets, which are clearly labeled.

The learning arrangement <NUM> may learn the model <NUM> from scenario data, e.g. accident reports, or from a supervision of drivers, e.g. from data in a Black box of a vehicle.

The modelling arrangement <NUM> may create the model <NUM> based on known relationships, e.g. according to the white box or the black box approach.

The selecting arrangement <NUM> may be adapted to determine the test case * depending on a result of a comparison of at least one individual probability σ1,. , σz with a threshold or with at least one other individual probability σ1,. In particular, the test case with the comparably higher probability or with the highest of all probabilities is selected. By way of example, when the probability σIII=<NUM>% and the probability σz=<NUM>%, the test case with the probability σIII is used for testing.

The selecting arrangement <NUM> may be adapted to provide the test case for testing or not depending on the result of an overlay of a context for the test case with at least one general condition for the principle PI,.

The selecting arrangement <NUM> may be adapted to provide the test case for testing or not depending on the result of a correlation analysis of the model <NUM>.

The selecting arrangement <NUM> is in one aspect adapted to determine a plurality of test cases, for testing whether the apparatus <NUM> operates according to a principle or not, from actions and contexts for the at least partially autonomous apparatus depending on a reference rule. The selecting arrangement <NUM> is in one aspect adapted to determine the plurality of test cases depending on a probabilistic network or computational intelligence. The term probabilistic network refers to a probabilistic inference computation that is based e.g. on a Bayesian network representing the model <NUM>. The term computational intelligence refers to a computation that is based on fuzzy logic type I or II, artificial neural networks or evolutionary computation representing the model <NUM>.

The selecting arrangement <NUM> may be adapted to determine the plurality of test cases depending on input from the rule-based model <NUM>, such as a fuzzy set model (type I or type II).

Alternatively, or additionally, the selecting arrangement <NUM> may be adapted to determine the plurality of test cases depending on input from the case-based reasoning database <NUM>.

The selecting arrangement <NUM> may be adapted to determine the test case * depending on the Bayesian network representing the model <NUM>.

In another aspect, the selecting arrangement <NUM> is adapted to determine the test case * from the plurality of test cases depending on the probability distribution σ1,.

The testing arrangement <NUM> is adapted to provide the test case * for testing at the at least partially autonomous apparatus <NUM>. The testing arrangement <NUM> is adapted to determine a result of the testing.

The test case is for example defined depending on at least one control signal for the at least partially autonomous apparatus <NUM> or a component thereof. The test case may additionally or alternatively be defined depending on at least one sensor signal, in particular an audio signal, a video signal, a radar sensor signal, an ultrasonic sensor signal or a time of flight sensor signal.

Exemplary test cases for testing an at least partial autonomous vehicle may include a video signal indicating an action wherein a person at a side of a road is moving into a lane of a road. In this case, a plurality of contexts may be defined by different environmental properties, e.g. illumination of the scenario at a daytime, nighttime or at dawn. Likewise, different obstacles hiding the person at least partially may be arranged in the video signal at different locations for different contexts.

For these test cases, a general rule may be defined, in particular by expert knowledge, indicating different reactions that are expected for different contexts of a scenario. For example, the general rule "WHEN detecting a person at a side of a road THEN reduce speed" may be defined. Other more specific rules such as correlations of several rules or rules that reduce a parameter space may be defined as well.

Exemplary test cases for testing the at least partial autonomous vehicle may include another video signal indicating a traffic sign at different scenarios. Additionally, to different illumination of the scenario the traffic sign may be at least partially obfuscated by snow or trees in different contexts.

An exemplary rule for these test cases may be a general rule "WHEN detecting that an object is obfuscated at least partially THEN detect the object by its geometry".

For testing, the test case defines at least on action and context of a scenario for the testing. The testing arrangement <NUM> is adapted to provide the context for the testing. The testing arrangement <NUM> is adapted to determine a result of the testing depending on a comparison between the action and the reaction.

The testing arrangement <NUM> may be adapted to capture the reaction at the at least partially autonomous apparatus.

For self-learning, the learning arrangement <NUM> may be adapted to capture a reaction to the context at the at least partially autonomous apparatus and learn the model <NUM> accordingly.

A method for test case generation is described with reference to <FIG>.

The method includes aspects of self-learning of the model <NUM> that are optional in case the model has been previously trained.

In a step <NUM>, scenarios for test cases and principles are provided. The scenarios and/or the principles may be labeled. The principles comprise a digital representation of one or more of a law, an accident report, a log, or human expertise or a combination thereof.

The input for determining the scenarios is for example a digital representation of Laws, accident reports, logs, human expertise.

These digital representations may be machine interpretable rules.

A linear independent Eigenspace of scenario vectors may be determined for the scenarios. A scenario vector SV defines for example correlations of actions a1,. , an and contexts K1,.

The scenario determination may comprise classification and clustering of scenarios that are suitable for testing a principle. The scenarios may be classified or clustered with statistical methods towards linear independent environmental variables.

A similarity between two scenarios may be determined. In this case, only one of the two scenarios may be used when a similarity between the two scenarios is detected. Otherwise, both scenarios may be used.

A prompt for user input may be determined in order to determine the similarity between two scenarios. In this case, only one of the scenarios may be used when a similarity between the two scenarios is detected from user input. Otherwise, both scenarios may be used.

In step <NUM> at least one rule for test case generation is determined from the scenarios and the principles.

In the step <NUM> the at least one rule is determined for correlating actions a1,. , an and contexts K1,. The at least one rule is determined depending on at least one of the scenarios. The rule defines for example correlations of classes of actions and classes of contexts independent from the executable actions and contexts.

The at least one rule may be determined for each of the scenario vectors in the linear independent Eigenspace of scenario vectors. This means, the rule defines what response is expected at the apparatus <NUM>. For example, for a person in front of a car the rule defines as reaction braking. A risk assignment of scenario vectors may be determined depending on laws and human expertise. A neural network, a fuzzy model (type I or type II) or a Bayesian networks may be determined based on probability and impact factors. For example, for some rule sun and rain together have low probability but have a high impact for testing a principle.

Step <NUM> comprises determining, storing and/or outputting the model <NUM> for test case generation depending on the at least one rule.

The model <NUM> may be determined depending on the rules according to a modeling goal. The modeling goal is achieved when all principles and scenarios are covered.

The goal is achieved by determining links in the model for all rules for all scenarios and all principles to define how the at least partially autonomous apparatus or user thereof should react. A general rule may be determined to define an expected reaction for a scenario.

The general rule may be determined depending on a rule-based model, in particular a fuzzy set model (type I or type II).

Potential reactions for a car include e.g. braking when a person is detected in front of the car. A risk assignment of scenario vectors, SV, with an artificial neural network, a fuzzy logic (type I or type II) or a Bayesian network may provide the general rule based on probability and impact factors. For example, sun and rain together result in a low probability but high impact.

The output of the model <NUM> is for example a risk matrix mapping scenario vectors, SV, and principles, PR, to test cases TC according to a risk evaluation for the combinations thereof.

The model <NUM> can be a self-learned model or be previously defined depending on expert knowledge.

Afterwards an optional step <NUM> is executed.

In the step <NUM> a plurality of test cases is determined according to a selection goal. The selection goal achieved when effective test strategies are selected. In the example, a test case TC is defined by the model <NUM> depending on the rule.

The input for selecting the plurality of test cases in the example is the risk matrix mapping scenario vectors, SV, and principles PR to the test cases TC according to the risk evaluation.

The step <NUM> may comprise evaluating an applicability of scenarios to the at least partially autonomous apparatus, to an architecture or a structure thereof, and may comprise identifying appropriate test cases. a dynamic test is identified for a component change instead of a full system test. A test case in this context may be defined by the model <NUM> depending on the rule.

The output may be a minimum effective test case matrix including test cases TC.

In the optional step <NUM>, the testing arrangement <NUM> executes the test cases TC with the goal of transparency of the results reporting.

The input may be the minimum effective test matrix including the test cases TC.

The test cases TC are applied in the example from the minimum effective test matrix to the apparatus or a subsystem thereof. The results are determined and reported in an intelligible way, to make artificial intelligence and machine learning behaviors transparent e.g. to engineers, safety experts, policy makers.

The output of the testing may be an intelligible test and defect report with scenario towards test results, TR, versus expected outcome. The output may comprise the feedback to previous steps for supervised optimization. For example a feedback for closed loop supervised learning and improvement may be provided.

Afterwards the method ends or continues for further scenarios and principles with step <NUM> until the learning goal is achieved. The learning goal is for example achieved, when the scenarios are complete.

Exemplary scenarios for testing are depicted in <FIG> depicts the correlation of the scenarios to situations and a system under test (SUT).

A test-scenario is in the example a multi-dimensional mapping of linear independent Eigen-Values of external situations, combined with a selection of internal parameters depending on a test strategy and architecture of the SUT. This means a scenario is a function f(situation, SUT). The test-scenarios may be represented by the linear independent Eigenspace of scenario vectors (SV).

In <FIG> the following internal parameters are depicted at columns of a two-dimensional exemplary representation of the multi-dimensional mapping:.

The following exemplary external situations are depicted in <FIG> by way of a knowledge graph syntax grouping selectable elements hierarchically, where one or more elements between a pair of parenthesis with the lowest hierarchical level of each example form a group comprising elements that are selectable individually as external situation to define the scenario:.

A resulting scenario is for example synthesized in a signal or signals for testing the at least partially autonomous apparatus <NUM>. Sensors at the at least partially autonomous apparatus <NUM> capture a response, i.e. reaction. The response may be used to learn a rule for actions or contexts depending on the leaves of the tree.

For testing the apparatus <NUM> or a component thereof, the at least partially autonomous apparatus <NUM> is presented actions in different contexts defined by the scenario. When the testing aims at validating that an operator of the at least partially autonomous apparatus <NUM> follows a certain rule, the at least partially autonomous apparatus <NUM> presents the operator with the actions in the contexts defined by the scenario and receives feedback from the operator.

Claim 1:
A device (<NUM>) for testing an at least partially autonomous apparatus or a component thereof, the device characterized by a learning arrangement (<NUM>) adapted to provide scenarios for test cases and principles to be tested, the principles comprising a digital representation of one or more of a law, an accident report, a log, or human expertise or a combination thereof, wherein the learning arrangement (<NUM>) is adapted to determine a plurality of rules for test case generation from the scenarios and the principles, and wherein the learning arrangement (<NUM>) is adapted to determine scenario vectors (SV) representing the scenarios, wherein each of the scenario vectors (SV) defines a rule for combining at least one action (a1, ..., an) with at least one context (K1, ..., Km), wherein the at least one action is an action that is presented to the at least partially autonomous apparatus or to the component thereof to prompt for a reaction, and wherein the at least one context refers to aspects of an environment and/or a situation in which the at least partially autonomous apparatus operates, and wherein a modelling arrangement (<NUM>) is adapted to determine, store and output a model (<NUM>) for test case generation depending on the plurality of rules.