Question: Not including Neidpath, who owned the other castle that Sir Simon Fraser had for most of its existence?

Context paragraphs:
Dean Valli: Dean Valli (born October 5, 1983 in North Vancouver, British Columbia) is an offensive lineman with the Canadian Football League's British Columbia Lions.  Valli attended Simon Fraser University and played for their football team, the Simon Fraser Clan.  He played four full seasons with the Clan from 2002 to 2005.  Dean graduated with a bachelor of arts from Simon Fraser University in 2006.  In 2005, Valli was named to the Canada West Universities Athletic Association all-star team.  He also represented SFU at the 2005 East West Bowl held in Waterloo, Ontario.  Valli was selected in the 1st round, 6th overall, by the Lions in the 2006 CFL draft.  2006 was his first year playing professional football.

Action at Happrew: The Battle of Happrew was a skirmish which took place around February 1304, during the First War of Scottish Independence.  A chevauchée of English knights, which included Robert de Clifford, William de Latimer and the later Scottish King, Robert the Bruce had been sent south from Dunfermline under Sir John Segrave to locate and capture the rebels Sir William Wallace and Sir Simon Fraser.

Simon Fraser Clan: The Simon Fraser Clan are the athletic teams that represent Simon Fraser University in Burnaby, British Columbia, Canada.  SFU's teams formerly played in the National Association of Intercollegiate Athletics of the United States for all sports.  In 1997, Simon Fraser sought to join the NCAA of the United States as a Division II school, but was turned down.  After this, SFU decided in 2000–01 to partially transfer to Canadian Interuniversity Sport (now U Sports).  Before the transfer, SFU did not compete in Canadian football, instead playing American football.

Fraser of Inverallochy: The Frasers of Inverallochy descend from Simon Fraser, 6th Lord Lovat (c. 1572–1633), and are one branch of the Clan Fraser of Lovat, who hail from Inverallochy.  Simon was Sheriff of Inverness, and married twice.  The Frasers of Lovat descend from this first marriage, while the Frasers of Inverallochy descend from a second marriage, through a son, Simon Fraser of Inverallochy.  This Simon Fraser, 1st of Inverallochy, had a greatgrandson, Charles Fraser, 6th of Inverallochy, who was created Lord Fraser of Muchall by James Francis Edward Stuart, known to Jacobites as King James VIII of Scotland, in the Jacobite Peerage of Scotland.

Simon Fraser Student Society: The Simon Fraser Student Society (SFSS) is the students' union of Simon Fraser University in Metro Vancouver, Canada.  It was founded after the opening of Simon Fraser University in 1967.

Oliver Castle: Oliver Castle was a medieval tower house, located in the upper Tweed Valley in the Scottish Borders.  The site of the hillfort known as Oliver Castle is to the north of the village of Tweedsmuir, although the site of the tower house is less certain.  Mentioned in a document of c.1200, it was originally part of the line of peel towers along the Tweed Valley.  It was replaced in the seventeenth century by a house, which was itself replaced in the late 18th century by the present Oliver House.  For most of its existence the property has been owned by members of the Tweedie family.

Simon Fraser (Australian politician): Sir Simon Fraser (21 August 1832 – 30 July 1919) was a Canadian-Australian politician, who was a member of the Australian Senate.

Terry Fox Field: Terry Fox Field is a multi-purpose sport field on Burnaby Mountain in Burnaby, British Columbia.  It serves as the home field of the Simon Fraser Clan for association football, Canadian football (before 2010 season) and American football (from 2010 season to 2013 season) matches.  Simon Fraser University's men's lacrosse club also plays their home matches on Terry Fox Field, and Simon Fraser University's Department of Recreation and Athletics regularly hosts sport camps on Terry Fox Field.

Stanley J. Shapiro: Stanley J. Shapiro (born 1934) is Professor Emeritus of Marketing at Simon Fraser University, British Columbia, Canada.  He has served as dean of McGill University's School of Management (1973–1979) and dean of Business Administration (1987–1997) for Simon Fraser University.  Shapiro published many books in areas ranging from marketing history to the financial dimensions of marketing management.  Prior to joining Simon Fraser University, Dr. Shapiro was a Professor of Marketing (1967–1980), and he had earlier taught at the Wharton School for three years before entering industry.  Dr. Shapiro has also been a Visiting Professor at Queen’s University, Bentley College and Royal Roads University.  After retirement, he taught one of the first entirely “online” MBA I level marketing courses.

Simon Fraser (d. 1306): Sir Simon Fraser of Oliver and Neidpath was a Scottish knight who fought in the Wars of Scottish Independence, for which he was hanged, drawn, and quartered in 1306.

Supporting paragraphs: Simon Fraser (d. 1306) Oliver Castle
Answer: the Tweedie family