No, you should not trim brown parts of the lavender plant completely. You can prune the branches but leave 2 inches of foliage for the plant to grow back.