I have one idea, how about writing a helper function to check frequencyDialer.setFrequency in which we can check the normal and edge cases? like this:

    function setFrequency(frequency) {
      if (frequency < mozFMRadio.frequencyLowerBound) {
        assert.equal(frequencyDialer.setFrequency(frequency), mozFMRadio.frequencyLowerBound);
      } else if(frequency > mozFMRadio.frequencyUpperBound) {
        assert.equal(frequencyDialer.setFrequency(frequency), mozFMRadio.frequencyUpperBound);
      } else {
        assert.equal(frequencyDialer.setFrequency(frequency), frequency);
      }
    }

and you can write the test like this:
 
    setFrequency(0);
    setFrequency(mozFMRadio.frequencyUpperBound + 10);
    setFrequency(mozFMRadio.frequencyLowerBound - 10);