A computer-implemented method for interaction-based language learning, the method comprising: encoding, at an encoding Recurrent Neural Network, encoding-RNN, at a time step, a natural language input comprising one or more words related to a visual image (202, 302) and an initial state into a state vector; producing, at a controller network (250, 400), an output control vector based on the state vector, the controller network (250, 400) comprising a residue control module and a Gaussian Policy module, wherein the residue control module is configured to add a content modifying vector to the state vector to generate a control vector, the content modifying vector being obtained by inputting the state vector to a transformer network, the transformer network being implemented as two fully-connected layers with Rectified Linear Unit, ReLU, activation, and wherein the Gaussian Policy module is configured to generate the output control vector with a Gaussian distribution conditioned on the control vector from the residue control module; obtaining a visual feature vector output from a visual encoder (210, 212, 300) by inputting the visual image (202, 302) to the visual encoder (210, 212, 300); generating, at an action-RNN, a response to the natural language input based on the output control vector and the visual feature vector; and generating feedback, comprising both a next natural language input in a next time step and a reward, according to both the natural language input and the generated response; wherein the reward is a scalar-valued feedback with positive value as encouragement and negative value as discouragement according to correctness of the response; wherein the encoding-RNN and the action-RNN share parameters; and wherein the controller network (250, 400) is trained using reinforcement learning based on the feedback including the reward, and the encoding-RNN is trained to predict the next natural language input conditioned on previous language inputs and the visual image. encoding, at an encoding Recurrent Neural Network, encoding-RNN, at a time step, a natural language input comprising one or more words related to a visual image (202, 302) and an initial state into a state vector; producing, at a controller network (250, 400), an output control vector based on the state vector, the controller network (250, 400) comprising a residue control module and a Gaussian Policy module, wherein the residue control module is configured to add a content modifying vector to the state vector to generate a control vector, the content modifying vector being obtained by inputting the state vector to a transformer network, the transformer network being implemented as two fully-connected layers with Rectified Linear Unit, ReLU, activation, and wherein the Gaussian Policy module is configured to generate the output control vector with a Gaussian distribution conditioned on the control vector from the residue control module; obtaining a visual feature vector output from a visual encoder (210, 212, 300) by inputting the visual image (202, 302) to the visual encoder (210, 212, 300); generating, at an action-RNN, a response to the natural language input based on the output control vector and the visual feature vector; and generating feedback, comprising both a next natural language input in a next time step and a reward, according to both the natural language input and the generated response; wherein the reward is a scalar-valued feedback with positive value as encouragement and negative value as discouragement according to correctness of the response; wherein the encoding-RNN and the action-RNN share parameters; and wherein the controller network (250, 400) is trained using reinforcement learning based on the feedback including the reward, and the encoding-RNN is trained to predict the next natural language input conditioned on previous language inputs and the visual image.