A scene-aware dialog system, comprising: an input interface configured to receive a sequence of video frames, contextual information, and a query; a memory configured to store at least one neural network comprising a visual scene-aware dialog neural network trained to generate a response to the input query by analyzing one or combination of input sequence of video frames and the input contextual information provided to the neural network; a processor configured to detect and classify objects in each video frame of the sequence of video frames; determine relationships among the classified objects in each of the video frame, wherein at least one video frame of the sequence of video frames includes at least two classified objects; extract features representing the classified objects and the determined relationships for each of the video frame to produce a sequence of feature vectors, wherein there is one feature vector for one video frame; and submit the sequence of feature vectors, the input query and the input contextual information to the neural network to generate a response to the input query; and an output interface to render the response to the input query; characterized in that the relationship between the two classified objects is an intra-frame object relationship confined within the video frame, the processor is further configured to integrate region of interests of objects in the sequence of video frames for determining relationships between two objects of the classified objects, and the region of interests is a union box of bounding boxes of the two objects of the classified objects. an input interface configured to receive a sequence of video frames, contextual information, and a query; a memory configured to store at least one neural network comprising a visual scene-aware dialog neural network trained to generate a response to the input query by analyzing one or combination of input sequence of video frames and the input contextual information provided to the neural network; a processor configured to detect and classify objects in each video frame of the sequence of video frames; determine relationships among the classified objects in each of the video frame, wherein at least one video frame of the sequence of video frames includes at least two classified objects; extract features representing the classified objects and the determined relationships for each of the video frame to produce a sequence of feature vectors, wherein there is one feature vector for one video frame; and submit the sequence of feature vectors, the input query and the input contextual information to the neural network to generate a response to the input query; and detect and classify objects in each video frame of the sequence of video frames; determine relationships among the classified objects in each of the video frame, wherein at least one video frame of the sequence of video frames includes at least two classified objects; extract features representing the classified objects and the determined relationships for each of the video frame to produce a sequence of feature vectors, wherein there is one feature vector for one video frame; and submit the sequence of feature vectors, the input query and the input contextual information to the neural network to generate a response to the input query; and an output interface to render the response to the input query; characterized in that the relationship between the two classified objects is an intra-frame object relationship confined within the video frame, the processor is further configured to integrate region of interests of objects in the sequence of video frames for determining relationships between two objects of the classified objects, and the region of interests is a union box of bounding boxes of the two objects of the classified objects.