The novel tells the story of Jude Fawley, who lives in a village in southern England (part of Hardy's fictional county of Wessex), who yearns to be a scholar at "Christminster", a city modelled on Oxford. As a youth, Jude teaches himself Classical Greek and Latin in his spare time, while working first in his great-aunt's bakery, with the hope of entering university. But before he can try to do this the na誰ve Jude is seduced by Arabella Donn, a rather coarse and superficial local girl who traps him into marriage by pretending to be pregnant. The marriage is a failure, and they separate by mutual agreement, and Arabella later emigrates to Australia, where she enters into a bigamous marriage. By this time, Jude has abandoned his classical studies.
After Arabella leaves him, Jude moves to Christminster and supports himself as a mason while studying alone, hoping to be able to enter the university later. There, he meets and falls in love with his free-spirited cousin, Sue Bridehead. But, shortly after this, Jude introduces Sue to his former schoolteacher, Mr. Phillotson, whom she eventually marries. However, she soon regrets this, because in addition to being in love with Jude, she is physically disgusted by her husband, and, apparently, by sex in general. Sue soon leaves Phillotson for Jude. Because of the scandal Phillotson has to give up his career as a schoolmaster.
Sue and Jude spend some time living together without any sexual relationship, because of Sue's dislike both of sex and the institution of marriage. Soon after, Arabella reappears and this complicates matters. But Arabella and Jude divorce and she legally marries her bigamous husband, and Sue also is divorced. However, following this, Arabella reveals that she had a child of Jude's, eight months after they separated, and subsequently sends this child to his father. He is named Jude and nicknamed "Little Father Time" because of his intense seriousness and moroseness.
Jude eventually convinces Sue to sleep with him and, over the years, they have two children together and expect the third. But Jude and Sue are socially ostracised for living together unmarried, especially after the children are born. Jude's employers dismiss him because of the illicit relationship, and the family is forced into a nomadic lifestyle, moving from town to town across Wessex seeking employment and housing before eventually returning to Christminster. Their socially troubled boy, "Little Father Time", comes to believe that he and his half-siblings are the source of the family's woes. The morning after their arrival in Christminster, he murders Sue's two children and commits suicide by hanging. He leaves behind a note that simply reads, "Done because we are too menny." Shortly thereafter, Sue has a miscarriage.
Beside herself with grief and blaming herself for "Little Father Time"'s actions, Sue turns to the church that she has rebelled against and comes to believe that the children's deaths were divine retribution for her relationship with Jude. Although horrified at the thought of resuming her marriage with Phillotson, she becomes convinced that, for religious reasons, she should never have left him. Arabella discovers Sue's feelings and informs Phillotson, who soon proposes they remarry. This results in Sue leaving Jude once again for Phillotson. Jude is devastated and remarries Arabella after she plies him with alcohol to once again trick him into marriage.
After one final, desperate visit to Sue in freezing weather, Jude becomes seriously ill and dies within the year. It is revealed that Sue has grown "staid and worn" with Phillotson. Arabella fails to mourn Jude's passing, instead setting the stage to ensnare her next suitor.
The events of Jude the Obscure occur over a 19-year period, but no dates are specifically given in the novel. Aged 11 at the beginning of the novel, by the time of his death, Jude seems much older than his thirty years, for he has experienced so much disappointment and grief in his total life experience. It would seem that his burdens exceeded his sheer ability to survive, much less to triumph.

Question: Why do Jude and Sue end up having to move around from place to place? 
Answer: They are socially ostracised because of their relationship, living together and being unmarried.