Choosing the dimensionality for word or sentence embeddings depends on the specific task, dataset size, and computational resources. Typically, a dimension between 100 to 300 for word embeddings and 300 to 768 for sentence embeddings, like BERT, is common. Experimentation and model performance evaluation are key in determining the ideal dimensionality for the task at hand.