websocket: fix checking payload_len safety check for fragmentation

- a crafted websocket frame  using values spanning max unsinged int could have
  caused a failed detection of unsupported fragmentation and invalid memory access
- thanks to Peter Dunkley and Hugh Waite for assisting and pin-pointing
  where everything happens