Chemical weathering attacks and alters the rock's minerals. Primarily it is the wind and water of this earth that cause most physical weathering and erosion. The process of shaking the sugar cube like the effect of wind and rain on a rock in the sense that there is an abrasive force rubbing against the cube, breaking off the tiny pieces. Wind, for example, is extremely powerful method of wearing down the surface of a rock. You can see and feel how this happens on a windy day.
Water also is a powerful erroding force.
Water is capable of lifting great weights and rubbing off layers of rock. The force of the water helps chip away the rock and floating pieces act as miniature hammers to break even more pieces off. Again, the harder the layer, the slower the process. So we get areas like the grand canyon where the river has always traveled the path of least resistance and broken down the weaker rocks first. Some rocks are harder than others.