A processor-implemented method for localized smoke removal and color restoration of a real-time video, the method comprising the steps of: receiving, via one or more hardware processors, the real-time video, wherein the real-time video comprises a plurality of video frames; identifying, via the one or more hardware processors, one or more smoky video frames and one or more smoke-free video frames, for the real-time video, from the plurality of video frames, using a smoke video frame identification model, wherein each smoky video frame of the one or more smoky video frames comprises one or more precise smoke regions; generating, via the one or more hardware processors, a smoky feature map for each smoky video frame of the one or more smoky video frames, using smoke relevant features of the corresponding smoky video frame, wherein the smoky feature map for each smoky video frame comprises features of the one or more precise smoky regions; automatically estimating, via the one or more hardware processors, an air-light for each smoky video frame of the one or more smoky video frames, using the corresponding smoky feature map; generating, via the one or more hardware processors, an intermediate de-smoked video frame for each smoky video frame of the one or more smoky video frames, based on the corresponding estimated air-light, using a de-smoking algorithm; identifying, via the one or more hardware processors, a smoke-free reference video frame for each smoky video frame of the one or more smoky video frames, out of the one or more smoke-free video frames, wherein the smoke-free reference video frame for each smoky video frame is a smoke-free video frame out of the one or more smoke-free video frames, present just before the corresponding smoky video frame; generating, via the one or more hardware processors, a de-smoked video frame for each smoky video frame of the one or more smoky video frames, by compensating color information obtained from the corresponding smoke-free reference video frame, locally in the one or more precise smoke regions of the corresponding intermediate de-smoked video frame; and stitching, via the one or more hardware processors, the de-smoked video frame for each smoky video frame of the one or more smoky video frames, to obtain a de-smoked video in the real-time. receiving, via one or more hardware processors, the real-time video, wherein the real-time video comprises a plurality of video frames; identifying, via the one or more hardware processors, one or more smoky video frames and one or more smoke-free video frames, for the real-time video, from the plurality of video frames, using a smoke video frame identification model, wherein each smoky video frame of the one or more smoky video frames comprises one or more precise smoke regions; generating, via the one or more hardware processors, a smoky feature map for each smoky video frame of the one or more smoky video frames, using smoke relevant features of the corresponding smoky video frame, wherein the smoky feature map for each smoky video frame comprises features of the one or more precise smoky regions; automatically estimating, via the one or more hardware processors, an air-light for each smoky video frame of the one or more smoky video frames, using the corresponding smoky feature map; generating, via the one or more hardware processors, an intermediate de-smoked video frame for each smoky video frame of the one or more smoky video frames, based on the corresponding estimated air-light, using a de-smoking algorithm; identifying, via the one or more hardware processors, a smoke-free reference video frame for each smoky video frame of the one or more smoky video frames, out of the one or more smoke-free video frames, wherein the smoke-free reference video frame for each smoky video frame is a smoke-free video frame out of the one or more smoke-free video frames, present just before the corresponding smoky video frame; generating, via the one or more hardware processors, a de-smoked video frame for each smoky video frame of the one or more smoky video frames, by compensating color information obtained from the corresponding smoke-free reference video frame, locally in the one or more precise smoke regions of the corresponding intermediate de-smoked video frame; and stitching, via the one or more hardware processors, the de-smoked video frame for each smoky video frame of the one or more smoky video frames, to obtain a de-smoked video in the real-time.