A computer-implemented method for interaction-based language learning, the method comprising: encoding , at an encoding Recurrent Neural Network, encoding-RNN , at a time step, a natural language input comprising one or more words related to a visual image and an initial state into a state vector; producing , at a controller network , an output control vector based on the state vector, the controller network comprising a residue control module and a Gaussian Policy module , wherein the residue control module is configured to add a content modifying vector to the state vector to generate a control vector, the content modifying vector being obtained by inputting the state vector to a transformer network, the transformer network being implemented as two fully-connected layers with Rectified Linear Unit, ReLU, activation, and wherein the Gaussian Policy module is configured to generate the output control vector with a Gaussian distribution conditioned on the control vector from the residue control module ; obtaining a visual feature vector output from a visual encoder by inputting the visual image to the visual encoder ; generating , at an action-RNN , a response to the natural language input based on the output control vector and the visual feature vector; and generating feedback, comprising both a next natural language input in a next time step and a reward, according to both the natural language input and the generated response; wherein the reward is a scalar-valued feedback with positive value as encouragement and negative value as discouragement according to correctness of the response; wherein the encoding-RNN and the action-RNN share parameters; and wherein the controller network is trained using reinforcement learning based on the feedback including the reward, and the encoding-RNN is trained to predict the next natural language input conditioned on previous language inputs and the visual image. encoding , at an encoding Recurrent Neural Network, encoding-RNN , at a time step, a natural language input comprising one or more words related to a visual image and an initial state into a state vector; producing , at a controller network , an output control vector based on the state vector, the controller network comprising a residue control module and a Gaussian Policy module , wherein the residue control module is configured to add a content modifying vector to the state vector to generate a control vector, the content modifying vector being obtained by inputting the state vector to a transformer network, the transformer network being implemented as two fully-connected layers with Rectified Linear Unit, ReLU, activation, and wherein the Gaussian Policy module is configured to generate the output control vector with a Gaussian distribution conditioned on the control vector from the residue control module ; obtaining a visual feature vector output from a visual encoder by inputting the visual image to the visual encoder ; generating , at an action-RNN , a response to the natural language input based on the output control vector and the visual feature vector; and generating feedback, comprising both a next natural language input in a next time step and a reward, according to both the natural language input and the generated response; wherein the reward is a scalar-valued feedback with positive value as encouragement and negative value as discouragement according to correctness of the response; wherein the encoding-RNN and the action-RNN share parameters; and wherein the controller network is trained using reinforcement learning based on the feedback including the reward, and the encoding-RNN is trained to predict the next natural language input conditioned on previous language inputs and the visual image. <EOS>