Baleen whales feed mostly on krill and small fish.
If the abundance of these organisms goes down for some reason, whatever that reason might be, whales would certainly have a hard time. Lets say, for example, that the abundance of Antarctic Krill went down because humans started to harvest the krill more. Almost certainly, the populations of the whales would go down too. Whales may be able to switch their diet concentration to another species of prey, but this would probably only occur after the population has already suffered an initial decline. The new prey species would probably not be able to support whale populations as they are now.