Fix some bugs and misbevhaviour in custom_optimize()

* The timeout is now cancelled when the optimization is already
  finished.

* Handle the case where the optimization finished without finding any
  paramters.