Fix not properly handling situations when we read frame fragments