A processor implemented method for video analysis, comprising: receiving a video data as input for analysis, via one or more hardware processors; splitting the video data to a plurality of frames of fixed length, via the one or more hardware processors; capturing temporal information from a sequence of frames from the plurality of frames, by processing the sequence of frames using a three dimensional convolution unit among a plurality of three dimensional convolution units in a three dimensional convolution unit network, via the one or more hardware processors; capturing spatial information for each frame by processing each frame using a 2-Dimensional,2D, convolution unit and a region proposal unit, via the one or more hardware processors; feeding the temporal information and the spatial information to a corresponding Long Short Term Memory,LSTM, block from among a plurality of LSTM blocks in a first LSTM layer, via the one or more hardware processors; generating a latent representation of the video as output, by the first LSTM layer, via the one or more hardware processors; extracting micro-scale information comprising a plurality of hidden state representations and cell state representations of the video, from the latent representation of the video, using an attention network, via the one or more hardware processors; feeding the micro-scale information as input to a second LSTM layer, via the one or more hardware processors; extracting spatio-temporal information corresponding to the collected video, by processing the micro-scale information using the second LSTM layer, via the one or more hardware processors; and training a neural network using the extracted spatio-temporal information, via the one or more hardware processors. receiving a video data as input for analysis, via one or more hardware processors; splitting the video data to a plurality of frames of fixed length, via the one or more hardware processors; capturing temporal information from a sequence of frames from the plurality of frames, by processing the sequence of frames using a three dimensional convolution unit among a plurality of three dimensional convolution units in a three dimensional convolution unit network, via the one or more hardware processors; capturing spatial information for each frame by processing each frame using a 2-Dimensional,2D, convolution unit and a region proposal unit, via the one or more hardware processors; feeding the temporal information and the spatial information to a corresponding Long Short Term Memory,LSTM, block from among a plurality of LSTM blocks in a first LSTM layer, via the one or more hardware processors; generating a latent representation of the video as output, by the first LSTM layer, via the one or more hardware processors; extracting micro-scale information comprising a plurality of hidden state representations and cell state representations of the video, from the latent representation of the video, using an attention network, via the one or more hardware processors; feeding the micro-scale information as input to a second LSTM layer, via the one or more hardware processors; extracting spatio-temporal information corresponding to the collected video, by processing the micro-scale information using the second LSTM layer, via the one or more hardware processors; and training a neural network using the extracted spatio-temporal information, via the one or more hardware processors.