Differences in Response Accuracy and Speed between FP32, 16, 8?
#73
by
elligottmc
- opened
I'm looking to purchase hardware and obviously a big leap from an A/L40 to an A100. But I don't want to try to cut corners and not have what I need to achieve my objectives. What differences in accuracy, speed or anything else can one expect when running Starcoder at FP32 versus 16? Same question 32 versus 16. Same question 16 versus 8. Insights from those experienced much appreciated!