Ideally, a 4 lb lamb should be enough for 6 people. It sounds like a lot, but meat shrinks during cooking. So this 4 lb will be a little over 3 lb after cooking. Personally, I think it is better to have more than less meat. There is so much you can do with leftovers.