Initialize the video decoder before the network connection
Try using a frame parser to assemble packets into frames before sending them to the decoder.